Section 7(a)(2) of the US Endangered Species Act directs federal agencies to consult with FWS and/or NMFS if an action they fund, permit, or carry out may affect ESA-listed species.

The great majority of formal consultations do not have take recorded.

The dataset we’re using includes 6982 formal consultations for 826 species. The effects determinations are made on a consultation-by-species basis, which results in 12,569 records.
The first thing we see is that only 1,836 records (14.6%) have take recorded. Not all species in a consulation are likely to be adversely affected (LAA), and for those species there would be no take statement (n = 4,807). Using this smaller number as the denominator, take has been recorded for only 36.1% of records that should have take recorded. Simply recording the take authorized is fundamental to being able to tally how much has been allowed in total.
Notice that take was recorded in 103 instances in which the recorded determination was not LAA. This shows that the data in TAILS contains inconsistencies that should be addressed.

Whether or not take is recorded is strongly contingent on the office doing the consultation.

In a previous working paper we found that whether or not consultations had geographic coordinates for an action depended strongly on the field office doing the consultation. We see a similar pattern here. Some offices, like the New England and the Alabama Ecological Services Field Offices, have more records with take recorded than not. In contrast, offices including South Florida, Utah, Columbus, and Ventura do many formal consultations but for less than 5% can we tell the amount of authorized take.
The plot at left is sorted in descending order by the proportion of records with take recorded.

In addition to the spatial context of recording take, there is a temporal component.

Most field offices have so few instances of recording take that there is no pattern. But the Sacramento office - which recorded take more than any other office - shows a strong decline in the rate at which they recorded take. The Washington (state) field office peaked in recording take in 2010, but has since tapered off. We hypothesize that folks in the field may have realized that, because the take data is unstructured, it was too difficult to make use of.
Hover over squares to get values, and note that you can zoom in to particular sections by clicking and dragging a box.

The common words of recorded take might tell us a bit about the structure or content of take statements.

FWS recently formalized their practice of using proxies of numbers of individuals in ITSs, such as describing take in terms of acres of habitat. That this has been their practice shows up readily in the word cloud, with ‘habitat’ being the most-frquent word in the take data (reinforced with high-weighted terms like ‘acre(s)’). Also notice the prominence of ‘harassment’: some significant portion of ITSs are recognizing non-lethal forms of take.
The word cloud is based on the frequency of the top 100 words, minus common words (e.g., stopwords), numbers, and the terms ‘incidental’ and ‘take’, which are relatively uninformative. 45 records include only a number (no units and no additional context) for take.

What can we learn about take authorized for the species with the most recorded take, bull trout?

The short answer is that there’s not a whole lot that we can learn about previously authorized take, even for the species that has had take recorded (n = 231) more than any other species. The most-used take entries (top-left) are wholly insufficient. (How can there be adverse effects with no take when ‘harm’, a component of take, can be defined as an adverse effect?) A decent number of take descriptions use measurable units in their text, but a brief perusal of the data indicates that the linear distance matches are often describing a distance from the action, not an amount lost.

Conclusion

Although we cannot estimate the total amount of take that has been authorized for any species using the data FWS has recorded in the TAILS database, we can learn from this data.

  • First and foremost, the Service needs to require that take is recorded for formal consultations. (And that goes for NE/NLAA determinations too; the take field should be ‘Not Applicable’.) Without data there is nothing that can possibly be analyzed.

  • Second, the Service needs to provide detailed guidance as to how take is recorded. Free-form descriptions will always be sub-par, and details of acres, numbers, feet, duration of effects, and so-forth are needed.

  • Third, and where the rubber meets the road, the Service needs to train and require personnel to use take data when evaluating the environmental baseline in consultations and when conducting five-year status reviews.

Without knowing how much is being ‘given away’ under section 7, the Service cannot know when to apply the brakes on authorized take. And without that knowledge, they are only making it harder to recover species that are approaching the brink through completely legal losses. 
---
title: "Section 7 Take"
author: "Jacob Malcom"
output: 
  flexdashboard::flex_dashboard:
    storyboard: true
    navbar:
      - { title: "Working papers", href: "https://defend-esc-dev.org/working_papers", align: right }
      - { title: "ESC Dev.", href: "https://defend-esc-dev.org/", align: right }
    social: menu
    source: embed
    theme: yeti
    css: _styles/ROAR_dash_style.css
---


```{r setup, include=FALSE}
library(d3wordcloud)
library(dplyr)
library(DT)
library(flexdashboard)
library(highcharter)
library(plotly)
library(stringr)
library(tm)
library(viridisLite)
load("_data/take_data.RData")
```

### Section 7(a)(2) of the US Endangered Species Act directs federal agencies to consult with [FWS](https://www.fws.gov) and/or [NMFS](http://www.nmfs.noaa.gov/) if an action they fund, permit, or carry out may affect ESA-listed species.

```{r}
sample <- head(full[full$with_take == 1, ], 25)
DT::datatable(sample, options = list(pageLength = 25))
```

***

##### If an action is likely to adversely affect (LAA) one or more ESA-listed species (i.e., 'formal consultation' is required), then FWS or NMFS may provide in their Biological Opinion an Incidental Take Statement (ITS) that stipulates the kind and amount of [take](http://www.nmfs.noaa.gov/pr/pdfs/laws/esa_section3.pdf) that is expected and authorized for the action. FWS records data on their consultations in the [TAILS database](http://www.pnas.org/content/112/52/15844.abstract), and one field of that database is 'take'. Tracking take is **critical** to understanding how much habitat and how many individuals are allowed to be lost through completely legal means. We have not previously analyzed the take field because that data is unstructured, but here we examine the take data more closely. 

> ##### The table at left is the first 25 rows of the data we're examining where take was recorded. But as we'll see, this is only a small selection of the consultations. And we'll also see more problems like the take record in the very first row of the table (simply "4"...4 what?).

### The great majority of formal consultations do not have take recorded.

```{r, cache = TRUE}
    with_det <- regexpr(full$BO_determ, pattern = "[Jj]eop|Modifi|Formal Consul")

    full$with_det <- ifelse(!is.na(with_det) & with_det != -1,
                            1,
                            0)

    cols <- viridis(2)
    cols <- substr(cols, 0, 7)
    
    cur_dat <- table(full$with_det, full$with_take)
    colnames(cur_dat) <- c("no take recorded", "take recorded")
    row.names(cur_dat) <- c("no LAA determination", "LAA determination")
    aplot <- plot_ly(x = colnames(cur_dat), 
                     y = cur_dat[1,], 
                     type = "bar",
                     name = "no LAA",
                     marker = list(color=cols[1]))
    bplot <- add_trace(aplot, 
                       x = colnames(cur_dat), 
                       y = cur_dat[2,],
                       type = "bar",
                       name = "LAA",
                       marker = list(color=cols[2]))
    layout(bplot, barmode = "stack",
           title = "Records with take recorded and whether needed (LAA)",
           xaxis = list(title = ""), 
           yaxis = list(title = "# records"))
```

***

##### The dataset we're using includes 6982 formal consultations for 826 species. The effects determinations are made on a consultation-by-species basis, which results in 12,569 records.

##### The first thing we see is that only 1,836 records (14.6%) have take recorded. Not all species in a consulation are likely to be adversely affected (LAA), and for those species there would be no take statement (n = 4,807). Using this smaller number as the denominator, take has been recorded for only 36.1% of records that _should_ have take recorded. _Simply recording the take authorized is fundamental_ to being able to tally how much has been allowed in total. 

> ##### Notice that take was recorded in 103 instances in which the recorded determination was _not_ LAA. This shows that the data in TAILS contains inconsistencies that should be addressed.

### Whether or not take is recorded is strongly contingent on the office doing the consultation.

```{r, cache = TRUE}

    datt <- table(full$ESOffice, full$with_take)
    dat <- data.frame(office = row.names(datt),
                      without = datt[,1],
                      with = datt[,2])
    dat$ratio <- dat$with / (dat$with + dat$without)
    dat <- dat[order(-dat$ratio), ]

    cols <- viridis(2)
    cols <- substr(cols, 0, 7)

    highchart() %>% 
      hc_xAxis(categories = dat$office,
               title = list(text = "Field Office")) %>%
      hc_yAxis(title = list(text = "# records")) %>% 
      hc_add_series(name = "No", data = dat$without, type = "column") %>%
      hc_add_series(name = "Yes", data = dat$with, type = "column") %>%
      hc_title(text = "Records with and without take recorded by field office",
               margin = 20) %>%
      hc_legend(title = list(text = "Take recorded"),
                             layout = "vertical",
                             align = "left",
                             verticalAlign = "top") %>%
      hc_colors(cols) %>%
      hc_tooltip(crosshairs = TRUE, 
                 backgroundColor = "#FCFFC5",
                 split = FALSE,
                 shared = TRUE, 
                 borderWidth = 0) %>%
      hc_exporting(enabled = TRUE)
```

***

##### In a [previous working paper](https://defend-esc-dev.org/working_papers/coordinates.html) we found that whether or not consultations had geographic coordinates for an action depended strongly on the field office doing the consultation. We see a similar pattern here. Some offices, like the New England and the Alabama Ecological Services Field Offices, have more records with take recorded than not. In contrast, offices including South Florida, Utah, Columbus, and Ventura do many formal consultations but for less than 5% can we tell the amount of authorized take.

> ##### The plot at left is sorted in descending order by the proportion of records with take recorded.

### In addition to the spatial context of recording take, there is a temporal component.

```{r, cache = TRUE}

dat <- full[full$with_take == 1 & full$FY >= 2008, c(2, 3)]
datt = table(dat$FY, dat$ESOffice)
dat2 <- as.matrix(datt)

plot_ly(z = dat2,
        type = "heatmap",
        x = colnames(dat2),
        y = row.names(dat2),
        colors = viridis(7)) %>%
layout(title = "Take recorded by fiscal year and field office",
       xaxis = list(title = "", tickangle = 45), 
       yaxis = list(title = "Fiscal Year"), 
       legend = list(title = "# records"),
       margin = list(b = 250))

```

***

##### Most field offices have so few instances of recording take that there is no pattern. But the Sacramento office - which recorded take more than any other office - shows a strong decline in the rate at which they recorded take. The Washington (state) field office peaked in recording take in 2010, but has since tapered off. We hypothesize that folks in the field may have realized that, because the take data is unstructured, it was too difficult to make use of.

> ##### Hover over squares to get values, and note that you can zoom in to particular sections by clicking and dragging a box.

### The common words of recorded take might tell us a bit about the structure or content of take statements.

```{r, cache = TRUE}
    cols <- viridis(5, 1)
    cols <- substr(cols, 0, 7)
    
    terms <- Corpus(VectorSource(sample(full[full$with_take == 1, ]$take, 1000)))
    desc <- tm_map(terms, removePunctuation)
    desc <- tm_map(desc, function(x){ removeWords(x, stopwords()) })
    desc <- tm_map(desc, function(x){ removeWords(x, c("the", "incidental", "take")) })
    tdm <- TermDocumentMatrix(desc)
    tdm <- removeSparseTerms(tdm, 0.99)
    m <- as.matrix(tdm)
    v <- sort(rowSums(m), decreasing = TRUE)
    d <- data.frame(word = names(v), freq = v)
    d <- d %>% 
         tbl_df() %>%
         arrange(desc(freq)) %>% 
         head(100)
    
    words <- d$word
    freqs <- d$freq
    
    d3wordcloud(words, 
                freqs, 
                rotate.min = 0,
                rotate.max = 0,
                spiral = "rectangular",
                tooltip = TRUE)

```

***

##### FWS recently formalized their practice of using proxies of numbers of individuals in ITSs, such as describing take in terms of acres of habitat. That this has been their practice shows up readily in the word cloud, with 'habitat' being the most-frquent word in the take data (reinforced with high-weighted terms like 'acre(s)'). Also notice the prominence of 'harassment': some significant portion of ITSs are recognizing non-lethal forms of take. 

> ##### The word cloud is based on the frequency of the top 100 words, minus common words (e.g., [stopwords](https://en.wikipedia.org/wiki/Stop_words)), numbers, and the terms 'incidental' and 'take', which are relatively uninformative. 45 records include only a number (no units and no additional context) for take.

### What can we learn about take authorized for the species with the most recorded take, bull trout?

```{r, cache = TRUE}

sub <- full[full$BO_species == "Bull Trout (Salvelinus confluentus)" & full$with_take == 1, ]
sub$take <- tolower(sub$take)
top_take <- sort(table(sub$take), decreasing = TRUE)
top_take <- data.frame(take = names(top_take), count = as.vector(top_take))
top_take <- head(top_take, 12)

plot_a <- plot_ly(x = top_take$take,
                  y = top_take$count, 
                  type = "bar",
                  marker = list(color = substr(viridis(1), 0, 7))) %>%
          layout(xaxis = list(title = "", tickangle = 45), 
                 yaxis = list(title = "# instances"), 
                 margin = list(b = 200))

n_mile <- length(grep(x = sub$take, pattern = "mile|feet|ft\\."))
n_acre <- length(grep(x = sub$take, pattern = "acre|ac\\."))
n_indi <- length(grep(x = sub$take, pattern = "individual|ind\\."))
n_stress <- length(grep(x = sub$take, pattern = "stress|subleth"))
n_redd <- length(grep(x = sub$take, pattern = "redd"))
lens <- c(n_mile, n_acre, n_indi, n_stress, n_redd)
patt <- c("mile | feet", "acre | ac.", "individual | ind.", "stress | subleth", "redd")

plot_b <- plot_ly(x = patt,
                  y = lens,
                  type = "bar",
                  marker = list(color = substr(viridis(1), 0, 7))) %>%
          layout(xaxis = list(title = ""), 
                 yaxis = list(title = "# instances"), 
                 margin = list(b = 200))

afun <- function(x) {
    q <- strsplit(x, split = " - ")
    return(q[[1]][1])
}

sub$work_cat <- sapply(sub$work_type, FUN = afun)
top_work <- sort(table(sub$work_cat), decreasing = TRUE)
top_work <- data.frame(work = names(top_work), count = as.vector(top_work))
top_work <- head(top_work, 12)

plot_c <- plot_ly(x = top_work$work, 
                  y = top_work$count, 
                  type = "bar",
                  marker = list(color = substr(viridis(1), 0, 7))) %>%
          layout(xaxis = list(title = "", tickangle = 45), 
                 yaxis = list(title = "# instances"), 
                 margin = list(b = 150))

top_agency <- sort(table(sub$lead_agency), decreasing = TRUE)
top_agency <- data.frame(ag = names(top_agency), count = as.vector(top_agency))
top_agency <- head(top_agency, 5)

plot_d <- plot_ly(x = top_agency$ag, 
                  y = top_agency$count, 
                  type = "bar",
                  marker = list(color = substr(viridis(1), 0, 7))) %>%
          layout(xaxis = list(title = "", tickangle = 45), 
                 yaxis = list(title = "# instances"), 
                 margin = list(b = 150))

# subplot(plot_a, plot_b, plot_c, plot_d, margin = 0.05, nrows = 2) %>% 
#     layout(showlegend = FALSE)
subplot(plot_a, plot_b, margin = 0.05, nrows = 1) %>% 
    layout(title = "Top take statements and key words about take", 
           showlegend = FALSE)

```

***

##### The short answer is that there's not a _whole lot_ that we can learn about previously authorized take, even for the species that has had take recorded (n = 231) more than any other species. The most-used take entries (top-left) are wholly insufficient. (How can there be adverse effects with no take when 'harm', a component of take, can be defined as an adverse effect?) A decent number of take descriptions use measurable units in their text, but a brief perusal of the data indicates that the linear distance matches are often describing a distance from the action, not an amount lost. 


### Conclusion


Although we cannot estimate the total amount of take that has been authorized for any species using the data FWS has recorded in the TAILS database, we can learn from this data. 

- _First and foremost_, the Service needs to require that take is recorded for formal consultations. (And that goes for NE/NLAA determinations too; the take field should be 'Not Applicable'.) Without data there is nothing that can possibly be analyzed.

- _Second_, the Service needs to provide detailed guidance as to how take is recorded. Free-form descriptions will always be sub-par, and details of acres, numbers, feet, duration of effects, and so-forth are needed. 

- _Third, and where the rubber meets the road_, the Service needs to train and require personnel to use take data when evaluating the environmental baseline in consultations _and_ when conducting five-year status reviews. 

Without knowing how much is being 'given away' under section 7, the Service cannot know when to apply the brakes on authorized take. And without that knowledge, they are only making it harder to recover species that are approaching the brink through completely legal losses.