vignettes/vignette_BCGcalc.Rmd
vignette_BCGcalc.RmdThe BCGcalc package was created to enable users to
generate Biological Condition Gradient (BCG) model outputs for
macroinvertebrate data from freshwater wadeable streams in the Puget
Lowlands and Willamette Valley ecoregions (Stamp and Gerritsen 2018).
With modification, this R code can be used to generate outputs for other
BCG models as well.
This vignette covers the basics going from raw data to model results. Any of the code in this Vignette can be copied and pasted to an R session and it will produce the same results as shown here. Each section of code (gray box) is independent such that no other code needs to be run except what is in that section. Thus, there is some repetition of steps between sections.
No files are exported in the examples in the vignette. All “write” statements have been commented out. If you wish to save the output remove the “#” from before each line of code. Several different “write” functions were used in the examples. This was intentional such that intermediate files are output as TSV (tab-separated values) files and final results are output as CSV (comma-separated values) files. Both formats will open in Excel.
When applying the BCG, users should keep in mind that they can run any data through the model and get a result. However, if samples for the Puget Lowlands and Willamette Valley macroinvertebrate BCG model do not meet the criteria below, results should be interpreted with caution because they are outside the experience of this particular BCG model.
Size: wadeable streams with drainage areas ranging from 1 to 100 mi2
Geographic area: Puget Lowlands (eco code 2) and Willamette Valley (eco code 3) EPA level3 ecoregions
Stream type: freshwater, perennial; no unique habitats (such as springs and seeps)
Target number of organisms: 500-count (subsampled to 600 total individuals where needed)
Sampling area: ≥8 ft2
Level of taxonomic resolution: lowest practical level except for mites, which should be collapsed to the Order-level (Trombidiformes)
Collection gear: D-Frame kick-nets with 500-micrometer net mesh
Collection method: a targeted “riffle only” sampling scheme (like those used by King County and ODEQ) or WA ECY’s multi-habitat, ‘reach-wide’ sampling scheme
Collection period: July through October
Results should be interpreted with caution if they are flagged for any of the criteria listed in the ‘Flags’ section at the end of the vignette (e.g., brackish influence, extreme dominance by one or two taxa).
Version 1 of the Puget Lowlands and Willamette Valley macroinvertebrate BCG model has known limitations and will need further testing in coming years. The model should be regarded as a beta version, with potential for refinement over time as the model is used with new data.
Stamp, J. and J. Gerritsen. 2018. Calibration of the Biological Condition Gradient (BCG) for Macroinvertebrate Assemblages in Puget Lowland/Willamette Valley Freshwater Wadeable Streams. Prepared by Tetra Tech for the US EPA Office of Water, Office of Science and Technology and US EPA Region 10.
The package is hosted on GitHub (https://github.com/leppott/BCGcalc) and can be installed
using the following lines of code. It is necessary to install the
devtools package.
# Installing the BCGcalc library (with the vignette) from GitHub
library(devtools)
install_github("leppott/BCGcalc", force=TRUE, build_vignettes=TRUE)After the BCGcalc package has been installed running the following line of code will open the help file with links to all functions.
help(package="BCGcalc")When you install the BCGcalc R package, the following files are contained in the ‘extdata’ folder in the BCGcalc library folder:
TaxaMaster_Bug_BCG_PugLowWilVal – this Excel file contains the master taxa list from the Puget Lowlands and Willamette Valley BCG project, and associated attribute information (NonTarget, BCG attribute, Thermal_Indicator, FFG, Habit, LifeCycle, TolVal). This file is not referenced by the R code; rather its purpose is to help users create their own input data files.
Rules - Excel file that contains worksheets with the BCG rules that the R code references; the worksheet titled ‘BCG_PugLowWilVal_500ct’ is used for the 500-count model. The worksheet titled ‘BCG_PugLowWilVal_300ct’ is used for the (preliminary) 300-count model.
Data_BCG_PugLowWilVal – Excel file that can be used as a template for your input file. It contains data for the 678 samples that were in the Puget Lowlands and Willamette Valley BCG calibration dataset. Column headings for required fields are highlighted in green.
MetricNames – Excel file that contains a list of all the potential metrics that can be calculated with the BCGcalc R tool. This file is not referenced by the R code.
MetricFlags - if any of these criteria are not met, the sample is flagged to alerts users that the sample is unusual (e.g., brackish water, too few organisms, reduced sampling effort). Results for flagged samples should be interpreted with caution.
PL_WV_ThermalIndicator_20180326 – detailed information on how the thermal indicator designations were made for the Puget Lowlands and Willamette Valley, using temperature tolerance analyses and expert elicitation.
ExampleMunge_Slope – see Example, New Data section below
ExampleMunge_UnformatedData - see Example, New Data section below
There are a number of required fields for the input file. If any fields are missing the user will be prompted as to which are missing and the user can decide whether to continue or quit. If the user continues, the missing fields will be added but will be filled with zero or NA (as appropriate). Any metrics based on the missing fields will not be valid.
Required Fields:
SAMPLEID
Unique sample identifier
Valid entries: character or number, must be unique
TAXAID
Unique taxa identifier
Valid entries: character or number, must be unique
N_TAXA
Number of individuals
Valid entries: number
EXCLUDE
Non-unique/non-distinct taxa are excluded from richness metric calculations but are counted in the other metrics. Appendix B of the ‘BCGcalc_README_20180919’ Word document describes the Exclude Taxa Decision Criteria that was used during BCG model calibration.
Valid entries: TRUE or FALSE
Non-unique/non-distinct taxa should be entered as “TRUE”
NONTARGET
Non-target taxa are not part of the intended capture list; e.g., fish, herps, water column taxa. They are excluded from all metric calculations.
Valid entries: TRUE or FALSE.
NonTarget taxa should be entered as “TRUE”
INDEX_NAME
Name of the BCG rules worksheet in the ‘Rules’ file (in the ‘extdata’ folder) that the R code is referencing.
Valid entries for the Puget Lowlands/Willamette Valley model: BCG_PugLowWilVal_500ct or BCG_PugLowWilVal_300ct.
SITE_TYPE
Select which BCG model to apply: Low (lo) gradient (<1% NHD+ v2 flowline slope) or high (hi) gradient (≥ 1% NHD+ v2 flowline slope).
Valid entries: “hi” or “lo”.
PHYLUM, SUBPHYLUM, CLASS, ORDER, FAMILY, SUBFAMILY, TRIBE, GENUS
Phylogeny (see source file ‘TaxaMaster_Bug_BCG_PugLowWilVal’; note: this may not cover all of the taxa in your input file).
Other phylogenetic rankings (e.g., SubOrder or SuperFamily) can be included but are not used in the current metric calculations.
Valid entries: text
BCG_Attr
BCG attribute assignments for the Puget Lowlands and Willamette Valley (Stamp and Gerritsen 2018).
Valid entries: 1i, 1m, 2, 3, 4, 5, 6; if not available, leave blank or enter ‘NA’.
FFG, HABIT, LIFE_CYCLE, TOLVAL, THERMAL_INDICATOR
FFG
Valid values for FFG: CG, CF, PR, SC, SH
Function feeding group entries in the ‘TaxaMaster_Bug_BCG_PugLowWilVal’ file were compiled from ODEQ, WA ECY and the EPA National Aquatic Resource Surveys (NARS) but the BCG workgroup did not try to reach consensus.
HABIT
Valid values for HABIT: BU, CB, CN, SP, SW
Habit designations need to come from the user (the BCG workgroup did not attempt to reach regional consensus/reconcile differences across entities).
LIFE_CYLCE
Valid values for LIFE_CYCLE: UNI, SEMI, MULTI
Life cycle designations need to come from the user (the BCG workgroup did not attempt to reach regional consensus/reconcile differences across entities).
THERMAL_INDICATOR
Fields that are optional but encouraged: Area_mi2, SurfaceArea, Density_m2, Density_ft2. These fields are used for flagging unusual samples (see Flag section below). The R code can be adapted to include these fields in the output (see examples below).
Important!
BCG_Attr: For the BCG model results to be accurate and valid, users should make sure the BCG attributes in their input file match with those that are in the Excel file titled ‘TaxaMaster_Bug_BCG_PugLowWilVal.’ Several of the BCG rules are based on BCG attribute metrics.The BCG workgroup went through a lengthy process to reach consensus on the BCG attribute assignments (BCG_Attr) for the Puget Lowlands and Willamette Valley (for more details, see Stamp and Gerritsen 2018).
Site type: the designations in the BCG calibration dataset were based on the NHD+ v2 flowline slope but users can also base their gradient designations on other sources (such as reach-scale measurements). The 1% threshold should be regarded as a fuzzy versus distinct line, as it represents a transitional zone where streams likely share characteristics of both low and high gradient stream types. Because of this, we recommend that users run sites with 0.5% to 1.5% slope through both BCG models and report both sets of results.
Operational taxonomic units (OTUs): All benthic macroinvertebrate taxa should be identified to the appropriate operational taxonomic unit (OTU). For the Puget Lowlands/Willamette Valley BCG model, this is the lowest practical level except for mites, which should be collapsed to the Order-level (Trombidiformes).
Subsampling: The Puget Lowlands/Willamette Valley BCG model was calibrated for 500-count samples (+/- 20%). If any of your samples have more than 600 organisms, run your dataset through the Rarify (subsampling) routine (600 is +20% of the 500-count target), which is described below. The subsampling is done to make richness metrics comparable across the 500-count samples. Samples with less an 400 organisms will be flagged for small sample size and should be investigated further before accepting the model output.
The master taxa list is a data object in the package. It can be viewed or saved with the code below.
library(BCGcalc)
View(TaxaMaster_Ben_BCG_PugLowWilVal)
# Save to working directory
#write.csv(TaxaMaster_Ben_BCG_PugLowWilVal
# , "TaxaMaster_Ben_BCG_PugLowWilVal_20180927.csv")The first few lines are also displayed below.
| TaxaID | Phylum | SubPhylum | Class | SubClass | Order | SuperFamily | Family | Tribe | Genus | SubGenus | Species | BCG_Attr | NonTarget | Thermal_Indicator | Long_Lived | FFG | Habit | Life_Cycle | TolVal |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Oligochaeta | Annelida | Clitellata | Oligochaeta | Oligochaeta | NA | NA | NA | NA | NA | NA | NA | 4 | FALSE | NA | no | GC | CN | Uni | 7 |
| Trombidiformes | Arthropoda | Chelicerata | Arachnida | Acari | Trombidiformes | NA | NA | NA | NA | NA | NA | 4 | FALSE | NA | no | NA | CN | Uni | 7 |
| Elmidae | Arthropoda | Hexapoda | Insecta | Pterygota | Coleoptera | NA | Elmidae | NA | NA | NA | NA | 4 | FALSE | NA | yes | GC | CN | Uni | 7 |
| Lara | Arthropoda | Hexapoda | Insecta | Pterygota | Coleoptera | NA | Elmidae | NA | Lara | NA | NA | 4 | FALSE | cold_cool | yes | SH | CN | Uni | 7 |
| Optioservus | Arthropoda | Hexapoda | Insecta | Pterygota | Coleoptera | NA | Elmidae | NA | Optioservus | NA | NA | 4 | FALSE | cool_warm | yes | SC | CN | Uni | 7 |
| Heterlimnius corpulentus | Arthropoda | Hexapoda | Insecta | Pterygota | Coleoptera | NA | Elmidae | Elmini | Heterlimnius | NA | corpulentus | 3 | FALSE | cold_cool | yes | GC | CN | Uni | 7 |
The suite of functions in the BCGcalc package are
presented in two sections.
The first section will cover the core functions from metric calculation
to model results. The example will cover all steps in a single example.
Additional functions will be covered individually and each function will
have its own self contained example.
BCG Core Functions
A. Metric Calculation (BioMonTools package)
B. Metric Membership (Scoring)
C. Level Membership
D. Level Assignment
E. Add Flags
Additional (Optional) Functions
A. Subsample (BioMonTools package)
B. Metric Calculation, Save Specific Metrics
C. Flags (BioMonTools package)
Once your input file is formatted and ready to go, the next step is to run it through the R code. The R code generates BCG level assignments for each sample. Along the way it calculates the following: metric values, metric membership values, BCG level membership and BCG level assignments. It also adds flags to the BCG level assignment file.
Below are two examples of R code that can be used to generate BCG model outputs. This code is written for 500-count samples (which is what the PugLowWilVal BCG model is calibrated for).
The first example below is for test data that is automatically downloaded onto your computer when you install the BCGcalc R package. If it’s your first time running this code, and you are a novice R user, we recommend trying this code first (you can follow it verbatim, no edits needed, and it should generate the desired output).
The second example is for a ‘new’ data file (to simulate you running the R code on your own data).
The code below uses the test data that is contained in the package and is already of the proper format.
If you decide to run the example below, the following four files will appear in your working directory.
metrics.values.Test.tsv
Metric.Membership.Test.tsv
Level.Membership.Test.tsv
Levels.Flags.Test.csv
# Packages
library(BCGcalc)
library(readxl)
library(reshape2)
library(BioMonTools)
# Import
df.samps.bugs <- read_excel(
system.file("./extdata/Data_BCG_PugLowWilVal.xlsx",
package="BCGcalc"),
guess_max = 10^6)
# QC for TRUE/FALSE (both ok)
# Exclude to TRUE/FALSE
table(df.samps.bugs$Exclude)
# NonTarget to TRUE/FALSE
table(df.samps.bugs$NonTarget)
# Add missing columns
col_add_char <- c("INFRAORDER",
"HABITAT",
"ELEVATION_ATTR",
"GRADIENT_ATTR",
"WSAREA_ATTR",
"HABSTRUCT")
col_add_num <- "UFC"
df.samps.bugs[, col_add_char] <- NA_character_
df.samps.bugs[, col_add_num] <- NA_integer_
# 1.A. Calculate Metrics
# Extra columns to keep in results
keep.cols <- c("Area_mi2",
"SurfaceArea",
"Density_m2",
"Density_ft2")
# Run Function
df.metrics <- metric.values(df.samps.bugs,
"bugs",
fun.cols2keep = keep.cols)
# QC
dim(df.metrics)
# View(df.metrics)
# Save
# write.table(df.metrics,
# "Metric.Values.Test.tsv",
# col.names = TRUE,
# row.names = FALSE,
# sep = "\t")
# 1.B. Metric Membership
# Import Rules
df.rules <- read_excel(
system.file("./extdata/Rules.xlsx",
package = "BCGcalc"),
sheet = "Rules")
# Run function
df.Metric.Membership <- BCG.Metric.Membership(df.metrics,
df.rules)
# Show Results
# View(df.Metric.Membership)
# Save Results
# write.table(df.Metric.Membership,
# "Metric.Membership.Test.tsv",
# row.names = FALSE,
# col.names = TRUE,
# sep = "\t")
# 1.C. Level Assignment
# Run Function
df.Level.Membership <- BCG.Level.Membership(df.Metric.Membership,
df.rules)
# Show results
# View(df.Level.Membership)
# Save Results
# write.table(df.Level.Membership,
# "Level.Membership.Test.tsv",
# row.names = FALSE,
# col.names = TRUE,
# sep = "\t")
# 1.D. Level Membership
# Run Function
df.Levels <- BCG.Level.Assignment(df.Level.Membership)
# 1.E. Flags
# Import QC Checks
df.checks <- read_excel(system.file("./extdata/MetricFlags.xlsx",
package = "BCGcalc"),
sheet = "Flags")
# Run Function
df.flags <- qc.checks(df.metrics, df.checks)
# Change terminology; PASS/FAIL to NA/flag
df.flags[, "FLAG"][df.flags[, "FLAG"] == "FAIL"] <- "flag"
df.flags[, "FLAG"][df.flags[, "FLAG"] == "PASS"] <- NA
# long to wide format
df.flags.wide <- dcast(df.flags,
SAMPLEID ~ CHECKNAME,
value.var = "FLAG")
# Calc number of "flag"s by row.
df.flags.wide$NumFlags <- rowSums(df.flags.wide == "flag", na.rm = TRUE)
# Rearrange columns
NumCols <- ncol(df.flags.wide)
df.flags.wide <- df.flags.wide[, c(1, NumCols, 2:(NumCols-1))]
# Merge Levels and Flags
df.Levels.Flags <- merge(df.Levels,
df.flags.wide,
by.x = "SampleID",
by.y = "SAMPLEID",
all.x = TRUE)
# Show Results
# View(df.Levels.Flags)
# Summarize Results
table(df.flags[, "CHECKNAME"], df.flags[, "FLAG"], useNA = "ifany")
# Save Results
# write.csv(df.Levels.Flags, "Levels.Flags.Test.csv")The flexibility of R allows the user to get data from a multitude of
sources and then manipulate (munge) it to fit the format needed for the
functions in BCGcalc. The example below shows how to import
a file, add slope from a 2nd file, and to make a few changes to the data
in order to use it with the package. When the data source is fixed this
routine can be modified and then used on all future datasets to prepare
them for use with BCGcalc.
Before running this on your own data, you will need to update the directories, the name of the input file and potentially a few other fields. In this example, the input file is called ‘ExampleMunge_UnformatedData.xlsx’. Remember - when you type in your working directory that you either need two backslashes “\” or one forward slash “/”. If you copy and paste the directory from Windows File Explorer , it comes in as one backslash so you will need to manually correct this for the code to work.
# Setup
library(readxl)
library(dplyr)
library(BCGcalc)
library(BioMonTools)
library(reshape2)
# Read File
## FileName
fn.data <- system.file("./extdata/ExampleMunge_UnformatedData.xlsx"
, package="BCGcalc")
# wd <- "F:\\myDocs"
# fn.data <- file.path(wd, "ExampleMunge_UnformatedData.xlsx")
## Worksheet
sh.data <- "SamplesWithBioticAttributesAndR"
## Import
### set "guess" to a large number to avoid type being wrong
df.data <- read_excel(fn.data,
sheet = sh.data,
guess_max = 12000)
dim(df.data)
# Munge
## Col Names
### convert to upper case
names(df.data) <- toupper(names(df.data))
### Rename Columns (base R) [dplyr::rename not working]
names(df.data)[names(df.data) == "SAMPLE ID"] <- "SAMPLEID"
names(df.data)[names(df.data) == "TAXON"] <- "TAXAID"
names(df.data)[names(df.data) == "SAMPLE ID"] <- "SAMPLEID"
names(df.data)[names(df.data) == "QUANTITY SUBSAMPLING"] <- "N_TAXA"
#names(df.data)[names(df.data) == "HILSENHOFF BIOTIC TOLERANCE INDEX"] <- "TOLVAL"
# df.data <- df.data %>% rename("SAMPLEID"="SAMPLE ID"
# , "TAXAID"="TAXON"
# , "N_TAXA"="QUANTITY SUBSAMPLING"
# , "TOLVAL"="HILSENHOFF BIOTIC TOLERANCE INDEX"
# )
### Create columns
df.data$EXCLUDE <- !df.data$UNIQUE
df.data$NONTARGET <- df.data$`OUTSIDE PROTOCOL`
# df.data$FFG <- NA
# df.data$FFG[df.data$PREDATOR==TRUE] <- "PR"
# df.data$HABIT <- NA
# df.data$HABIT[df.data$CLINGER==TRUE] <- "CN"
# df.data$LIFE_CYCLE <- NA
df.data$SITE_TYPE <- NA
# df.data$BCG_ATTR <- NA
# df.data$THERMAL_INDICATOR <- NA
df.data$INDEX_NAME <- "BCG_PugLowWilVal_500ct"
df.data$SURFACEAREA <- df.data$`SURFACE AREA`
df.data$AREA_MI2 <- NA
df.data$DENSITY_M2 <- NA
df.data$DENSITY_FT2 <- NA
# Add slope (and then gradient for SiteType) from NHD+ v2
fn.slope <- system.file("./extdata/ExampleMunge_Slope.xlsx",
package="BCGcalc")
# fn.slope <- file.path(wd, "ExampleMunge_Slope.xlsx")
df.slope <- read_excel(fn.slope)
names(df.slope) <- toupper(names(df.slope))
# merge files
df.comb.slope <- merge(df.data,
df.slope,
by.x = "SITE CODE",
by.y = "SITE_CODE",
all.x = TRUE)
# QC (rows)
dim(df.data)
dim(df.comb.slope)
nrow(df.data) == nrow(df.comb.slope)
df.comb.slope$SITE_TYPE <- df.comb.slope$`SLOPE CATEGORY`
# Update Taxa Attributes from Master Taxa List in Package
df.taxamaster <- TaxaMaster_Ben_BCG_PugLowWilVal
names(df.taxamaster) <- toupper(names(df.taxamaster))
## Assume phylogenetic information is correct.
col.auteco <- c("TAXAID",
"BCG_ATTR",
"THERMAL_INDICATOR",
"LONG_LIVED",
"FFG",
"HABIT",
"LIFE_CYCLE",
"TOLVAL")
df.comb.slope.auteco <- merge(df.comb.slope,
df.taxamaster[, col.auteco],
by.x = "TAXAID",
by.y = "TAXAID",
all.x = TRUE)
nrow(df.comb.slope) == nrow(df.comb.slope.auteco)
# SiteClass to Index_Class
df.comb.slope.auteco$INDEX_CLASS <- df.comb.slope.auteco$SITE_TYPE
# Create Anlaysis File
col2keep <- c("SAMPLEID",
"INDEX_NAME",
"INDEX_CLASS",
"AREA_MI2",
"SURFACEAREA",
"DENSITY_M2",
"DENSITY_FT2",
"TAXAID",
"N_TAXA",
"EXCLUDE",
"NONTARGET",
"PHYLUM",
"SUBPHYLUM",
"CLASS",
"ORDER",
"FAMILY",
"SUBFAMILY",
"TRIBE",
"GENUS",
"FFG",
"HABIT",
"LIFE_CYCLE",
"TOLVAL",
"BCG_ATTR",
"THERMAL_INDICATOR")
df.samps.bugs <- as.data.frame(df.comb.slope.auteco[, col2keep ])
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Repeat code from previous example
# (with minor edits)
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# QC for TRUE/FALSE (both ok)
# Exclude to TRUE/FALSE
table(df.samps.bugs$EXCLUDE)
# NonTarget to TRUE/FALSE
table(df.samps.bugs$NONTARGET)
# 1.A. Calculate Metrics
# Extra columns to keep in results
keep.cols <- toupper(c("Area_mi2",
"SurfaceArea",
"Density_m2",
"Density_ft2"))
# Run Function
df.metrics <- metric.values(df.samps.bugs,
"bugs",
fun.cols2keep = keep.cols,
boo.Shiny = TRUE)
# QC
dim(df.metrics)
# View(df.metrics)
# Save
# write.table(df.metrics
# , "Metric.Values.New.tsv"
# , col.names = TRUE
# , row.names = FALSE
# , sep = "\t")
# 1.B. Metric Membership
# Import Rules
df.rules <- read_excel(system.file("./extdata/Rules.xlsx"
, package="BCGcalc")
, sheet="Rules")
names(df.rules)[names(df.rules) %in% "Index_Name"] <- "INDEX_NAME"
# Run function
df.Metric.Membership <- BCG.Metric.Membership(df.metrics, df.rules)
# Show Results
# View(df.Metric.Membership)
# Save Results
# write.table(df.Metric.Membership,
# "Metric.Membership.New.tsv",
# row.names = FALSE,
# col.names = TRUE,
# sep = "\t")
# 1.C. Level Assignment
# Run Function
df.Level.Membership <- BCG.Level.Membership(
df.Metric.Membership,
df.rules)
# # Show results
# View(df.Level.Membership)
# Save Results
# write.table(df.Level.Membership,
# "Level.Membership.New.tsv",
# row.names = FALSE,
# col.names = TRUE,
# sep = "\t")
# 1.D. Level Membership
# Run Function
df.Levels <- BCG.Level.Assignment(df.Level.Membership)
# 1.E. Flags
# Import QC Checks
df.checks <- read_excel(
system.file("./extdata/MetricFlags.xlsx",
package = "BCGcalc"),
sheet = "Flags")
# Run Function
df.flags <- qc.checks(df.metrics, df.checks)
# Change terminology; PASS/FAIL to NA/flag
df.flags[, "FLAG"][df.flags[, "FLAG"] == "FAIL"] <- "flag"
df.flags[, "FLAG"][df.flags[, "FLAG"] == "PASS"] <- NA
# long to wide format
df.flags.wide <- dcast(df.flags,
SAMPLEID ~ CHECKNAME,
value.var="FLAG")
# Calc number of "flag"s by row.
df.flags.wide$NumFlags <- rowSums(df.flags.wide == "flag", na.rm = TRUE)
# Rearrange columns
NumCols <- ncol(df.flags.wide)
df.flags.wide <- df.flags.wide[, c(1, NumCols, 2:(NumCols - 1))]
# Merge Levels and Flags
df.Levels.Flags <- merge(df.Levels,
df.flags.wide,
by.x = "SampleID",
by.y = "SAMPLEID",
all.x = TRUE)
# Show Results
# View(df.Levels.Flags)
# Summarize Results
table(df.flags[, "CHECKNAME"], df.flags[, "FLAG"], useNA = "ifany")
# Save Results
# write.csv(df.Levels.Flags, "Levels.Flags.New.csv")There are some ancillary tasks that are included in the
BCGcalc package for convenience.
The rarify function subsamples count data to a fixed
count per sample. It takes as an input a 3 column data frame (SampleID,
TaxonID, Count) and returns a similar dataframe with revised Counts. The
names of the columns does not matter as they are specified in the code.
Any non-count taxa (e.g., fish in a bug sample) should be removed prior
to using the rarify function. The function code is from
USEPA Corvallis John Van Sickle’s R code for RIVPACS (v1.0, 2005-06-10)
and was tweaked for the addition of a user provided seed so repeatable
results can be obtained. It is included in the package BioMonTools (https://github.com/leppott/BioMonTools).
The other function inputs are subsample size (target number of organisms in each sample) and seed. The seed is given so the results can be reproduced from the same input file. If no seed is given a random seed is used. An example seed is the date of admission to the Union for each state where the data is collected (e.g., Washington is 18891111). These values can be found on Wikipedia on the right sidebar for each State.
If you are running the 500-count BCG model and any of your samples have more than 600 organisms (the upper limit for the model), you should randomly subsample your data to 600 (600 is +20% of the 500-count target). This is done to make richness metrics comparable across the 500-count samples. You can do this with the Rarify routine.
Before you run the code below on your own data, you’ll need to update the directories, the name of the input file and potentially a few other fields. In this example, the input file is called ‘ExampleMunge_UnformatedData.xlsx’.
After you are done with the subsampling, bring the updated N_Taxa field into your input file (see example file titled ‘ExampleDataFile’; we retained the original data in an optional field called N_Taxa_orig). Then run your data file through the BCGcalc example code described previously in this document.
library(BCGcalc)
library(knitr)
library(BioMonTools)
# Subsample to 600 organisms (from over 600 organisms) for 12 samples.
## FileName
### Package example
df_data <- BioMonTools::data_bio2rarify
### Excel
# wd <- "F:\\myDocs"
# fn_data <- file.path(wd, "ExampleMunge_UnformatedData.xlsx")
# readxl::read_excel(fn.data)
### CSV
# fn_data <-
# df_data <- read.csv(fn_data)
#
df_biodata <- df_data
#dim(df_biodata)
#View(df_biodata)
# subsample
mySize <- 600
Seed_OR <- 18590214
Seed_WA <- 18891111
Seed_US <- 17760704
bugs_mysize <- BioMonTools::rarify(inbug = df_biodata,
sample.ID = "SampleID",
abund = "N_Taxa",
subsiz = mySize,
mySeed = Seed_US)
#dim(bugs.mysize)
#View(bugs.mysize)
# Compare pre- and post- subsample counts
df_compare <- merge(df_biodata,
bugs_mysize,
by=c("SampleID", "TaxaID"),
suffixes = c("_Orig","_600"))
df_compare <- df_compare[, c("SampleID",
"TaxaID",
"N_Taxa_Orig",
"N_Taxa_600")]
#View(df.compare)
# compare totals
tbl_compare <- head(df_compare)
tbl_compare_caption <- "First few rows of original and rarified data."
kable(tbl_compare, caption=tbl_compare_caption)| SampleID | TaxaID | N_Taxa_Orig | N_Taxa_600 |
|---|---|---|---|
| 01103CSR_Bug_2001-08-27_0 | Ceratopogoninae | 237 | 179 |
| 01103CSR_Bug_2001-08-27_0 | Cheumatopsyche | 2 | 0 |
| 01103CSR_Bug_2001-08-27_0 | Chironomini | 8 | 4 |
| 01103CSR_Bug_2001-08-27_0 | Cladocera | 2 | 2 |
| 01103CSR_Bug_2001-08-27_0 | Coenagrionidae | 2 | 1 |
| 01103CSR_Bug_2001-08-27_0 | Corbicula | 2 | 1 |
tbl_totals <- aggregate(
cbind(N_Taxa_Orig, N_Taxa_600) ~ SampleID,
df_compare, sum)
tbl_totals_caption <- "Comparison of total individuals per sample."
kable(tbl_totals, caption=tbl_totals_caption)| SampleID | N_Taxa_Orig | N_Taxa_600 |
|---|---|---|
| 01103CSR_Bug_2001-08-27_0 | 816 | 600 |
| 02087REF_Bug_2002-08-22_0 | 542 | 542 |
| 03013CSR_Bug_2003-07-01_0 | 508 | 508 |
| 03053CSR_Bug_2003-08-14_0 | 573 | 573 |
| 03054CSR_Bug_2003-08-18_0 | 558 | 558 |
| 06012CSR_Bug_2006-09-05_0 | 545 | 545 |
| 06019CSRw_Bug_2006-09-14_0 | 711 | 600 |
| 06021CSR_Bug_2006-08-01_0 | 706 | 600 |
| 06022CSR_Bug_2006-08-17_0 | 555 | 555 |
| 06024CSR_Bug_2006-07-27_0 | 569 | 569 |
| 06025CSR_Bug_2006-08-03_0 | 543 | 543 |
| 06027CSR_Bug_2006-07-17_0 | 569 | 569 |
# save the data
#write.table(bugs.mysize,paste("bugs",mySize,"txt",sep="."),sep="\t")You can adapt the code so that the metric calculation output only includes a subset of the metrics.
For example, you may only want to view the following metrics;
Metrics that go into the BCG model calculation (there are 12). See example #1 below.
Thermal indicator metrics (ti), c=cold, cc = cold/cool, cw =
cool/warm, w= warm.
See example #2 below.
You can select specific metrics one of two ways:
If you know the names of the metrics, write them into the
metric.values
function R code as shown in Example #1 below; or
Run the ‘normal’ metrics.values R code for to
generate an output that includes the full set of metrics, then examine
the results and run additional code as shown in Example # 2 to limit the
output to the desired metrics.
Knowing the names of the metrics, use them as input in the metric.values function.
# Packages
library(BCGcalc)
library(readxl)
library(knitr)
library(BioMonTools)
# Load Data
df.data <- read_excel(system.file("./extdata/Data_BCG_PugLowWilVal.xlsx"
, package="BCGcalc")
, guess_max = 10^6)
# Columns to keep
myCols <- c("Area_mi2",
"SurfaceArea",
"Density_m2",
"Density_ft2")
# Metrics of Interest (BCG)
col.met2keep <- c("ni_total",
"nt_total",
"nt_BCG_att1i2",
"pt_BCG_att1i23",
"pi_BCG_att1i23",
"pt_BCG_att56",
"pi_BCG_att56",
"nt_EPT_BCG_att1i23",
"pi_NonInsJugaRiss_BCG_att456",
"pt_NonIns_BCG_att456",
"nt_EPT",
"pi_NonIns_BCG_att456")
# Run Function
df.metval <- metric.values(df.data,
"bugs",
fun.cols2keep = myCols,
fun.MetricNames = col.met2keep,
boo.Shiny = TRUE)
# Select columns
col.ID <- c("SAMPLEID",
toupper(myCols),
"INDEX_NAME")
# Ouput
df.metval.bcg12 <- df.metval[, c(col.ID, col.met2keep)]
# RMD table
kable(head(df.metval.bcg12), caption = "Select Metrics, Example 1")Examine the results file and select certain metrics to keep.
# Packages
library(BCGcalc)
library(readxl)
library(knitr)
library(BioMonTools)
# Load Data
df.data <- read_excel(system.file("./extdata/Data_BCG_PugLowWilVal.xlsx"
, package="BCGcalc")
, guess_max = 10^6)
# Add missing columns
col_add_char <- c("INFRAORDER",
"HABITAT",
"ELEVATION_ATTR",
"GRADIENT_ATTR",
"WSAREA_ATTR",
"HABSTRUCT")
col_add_num <- "UFC"
df.data[, col_add_char] <- NA_character_
df.data[, col_add_num] <- NA_integer_
# Columns to keep
myCols <- c("Area_mi2",
"SurfaceArea",
"Density_m2",
"Density_ft2")
# Run Function
df.metval <- metric.values(df.data,
"bugs",
fun.cols2keep=myCols)
#> Joining with `by = join_by(SAMPLEID, INDEX_NAME, INDEX_CLASS)`
# Metrics of Interest
## thermal indicator (_ti_)
#names(df.metval)[grepl("_ti_", names(df.metval))]
col.met2keep <- c("ni_total",
"nt_total",
paste0("nt_ti_",
c("stenocold",
"cold",
"cool",
"warm",
"stenowarm")),
paste0("pi_ti_",
c("stenocold",
"cold",
"cool",
"warm",
"stenowarm")),
paste0("pt_ti_",
c("stenocold",
"cold",
"cool",
"warm",
"stenowarm"))
)
col.ID <- c("SAMPLEID",
toupper(myCols),
"INDEX_NAME")
# Ouput
df.metval.ci <- df.metval[, c(col.ID, col.met2keep)]
# RMD table
kable(head(df.metval.ci), caption = "Select Metrics, Example 2")| SAMPLEID | AREA_MI2 | SURFACEAREA | DENSITY_M2 | DENSITY_FT2 | INDEX_NAME | ni_total | nt_total | nt_ti_stenocold | nt_ti_cold | nt_ti_cool | nt_ti_warm | nt_ti_stenowarm | pi_ti_stenocold | pi_ti_cold | pi_ti_cool | pi_ti_warm | pi_ti_stenowarm | pt_ti_stenocold | pt_ti_cold | pt_ti_cool | pt_ti_warm | pt_ti_stenowarm |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 01103CSR_Bug_2001-08-27_0 | 44.084184 | NA | NA | NA | BCG_PugLowWilVal_500ct | 589 | 27 | 0 | 1 | 10 | 16 | 0 | 0 | 40.237691 | 60.95076 | 32.59762 | 0 | 0 | 3.703704 | 37.03704 | 59.25926 | 0 |
| 02087REF_Bug_2002-08-22_0 | 1.791475 | NA | NA | NA | BCG_PugLowWilVal_500ct | 542 | 38 | 0 | 23 | 28 | 8 | 0 | 0 | 18.265683 | 20.66421 | 69.55720 | 0 | 0 | 60.526316 | 73.68421 | 21.05263 | 0 |
| 03013CSR_Bug_2003-07-01_0 | 9.613734 | 8 | NA | NA | BCG_PugLowWilVal_500ct | 507 | 16 | 0 | 3 | 5 | 6 | 0 | 0 | 2.958580 | 13.21499 | 70.41420 | 0 | 0 | 18.750000 | 31.25000 | 37.50000 | 0 |
| 03053CSR_Bug_2003-08-14_0 | 3.061723 | 8 | NA | NA | BCG_PugLowWilVal_500ct | 573 | 36 | 0 | 19 | 24 | 9 | 0 | 0 | 46.073298 | 58.46422 | 24.78185 | 0 | 0 | 52.777778 | 66.66667 | 25.00000 | 0 |
| 03054CSR_Bug_2003-08-18_0 | 71.249741 | 8 | NA | NA | BCG_PugLowWilVal_500ct | 558 | 13 | 0 | 4 | 4 | 4 | 0 | 0 | 6.989247 | 11.82796 | 80.28674 | 0 | 0 | 30.769231 | 30.76923 | 30.76923 | 0 |
| 06012CSR_Bug_2006-09-05_0 | 82.338353 | 8 | NA | NA | BCG_PugLowWilVal_500ct | 545 | 40 | 0 | 18 | 28 | 15 | 0 | 0 | 33.944954 | 75.96330 | 61.65138 | 0 | 0 | 45.000000 | 70.00000 | 37.50000 | 0 |
Results should be interpreted with caution if they are flagged for any of the criteria listed below. If you run the BCGcalc code on the Test data above, columns with flags will be added into the file with the Level Assignments.
The checks for hi and lo gradient are the same so only one set of checks is shown below.
#> Joining with `by = join_by(SAMPLEID, INDEX_NAME, INDEX_CLASS)`| Index_Name | INDEX_CLASS | Metric_Name | Symbol | Value | CheckName | BioMonTools_MetNam | Comment | SITE_TYPE | INDEX_REGION |
|---|---|---|---|---|---|---|---|---|---|
| BCG_PugLowWilVal_500ct | Hi | ni_total | > | 600.0 | individuals, Large | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | ni_total | < | 450.0 | individuals, small | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | Density_m2 | < | 500.0 | Low density (m2) | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | Density_ft2 | < | 47.0 | Low density (ft2) | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | Area_mi2 | < | 2.0 | catchment, small | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | Area_mi2 | > | 100.0 | catchment, Large | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | SurfaceArea | < | 8.0 | surface area, small | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | ni_brackish | > | 0.0 | brackish organisms present | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | ni_Ramello | >= | 10.0 | Ramellogammarus | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_500ct | Hi | pi_dom02 | >= | 50.0 | individuals, dominant 02, Large | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | ni_total | > | 360.0 | individuals, Large | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | ni_total | < | 240.0 | individuals, small | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | Density_m2 | < | 300.0 | Low density (m2) | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | Density_ft2 | < | 28.2 | Low density (ft2) | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | Area_mi2 | < | 2.0 | catchment, small | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | Area_mi2 | > | 100.0 | catchment, Large | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | SurfaceArea | < | 8.0 | surface area, small | FALSE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | ni_brackish | > | 0.0 | brackish organisms present | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | ni_Ramello | >= | 10.0 | Ramellogammarus | TRUE | NA | Hi | Hi |
| BCG_PugLowWilVal_300ct | Hi | pi_dom02 | >= | 50.0 | individuals, dominant 02, Large | TRUE | NA | Hi | Hi |
| NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
| NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
| NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
| NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
| SampleID | INDEX_NAME | INDEX_CLASS | L1 | L2 | L3 | L4 | L5 | L6 | Membership_Total | Membership_Total_QC | Primary_BCG_Level | Primary_Membership | Secondary_BCG_Level | Secondary_Membership | Membership_Diff | Membership_Close | Continuous_BCG_Level | BCG_Status | BCG_Status2 | NumFlags | brackish organisms present | catchment, Large | catchment, small | individuals, dominant 02, Large | individuals, Large | individuals, small | Low density (ft2) | Low density (m2) | Ramellogammarus | surface area, small |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 01103CSR_Bug_2001-08-27_0 | BCG_PugLowWilVal_500ct | Lo | 0 | 0.0000000 | 0.0000000 | 0.0000000 | 1.0000000 | 0.0 | 1 | PASS | 5 | 1.0000000 | NA | 0.0000000 | 1.0000000 | NA | 5.000000 | 5 | 5 | 1 | NA | NA | NA | flag | NA | NA | NA | NA | NA | NA |
| 02087REF_Bug_2002-08-22_0 | BCG_PugLowWilVal_500ct | Hi | 0 | 0.0129151 | 0.9870849 | 0.0000000 | 0.0000000 | 0.0 | 1 | PASS | 3 | 0.9870849 | 2 | 0.0129151 | 0.9741697 | NA | 2.987085 | 3 | 3 | 2 | NA | NA | flag | flag | NA | NA | NA | NA | NA | NA |
| 03013CSR_Bug_2003-07-01_0 | BCG_PugLowWilVal_500ct | Hi | 0 | 0.0000000 | 0.0000000 | 0.1000000 | 0.8000000 | 0.1 | 1 | PASS | 5 | 0.8000000 | 4 | 0.1000000 | 0.7000000 | NA | 5.000000 | 5 | 5 | 1 | NA | NA | NA | flag | NA | NA | NA | NA | NA | NA |
| 03053CSR_Bug_2003-08-14_0 | BCG_PugLowWilVal_500ct | Hi | 0 | 0.0000000 | 1.0000000 | 0.0000000 | 0.0000000 | 0.0 | 1 | PASS | 3 | 1.0000000 | NA | 0.0000000 | 1.0000000 | NA | 3.000000 | 3 | 3 | 0 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
| 03054CSR_Bug_2003-08-18_0 | BCG_PugLowWilVal_500ct | Lo | 0 | 0.0000000 | 0.0000000 | 0.0000000 | 0.5000000 | 0.5 | 1 | PASS | 5 | 0.5000000 | 6 | 0.5000000 | 0.0000000 | tie | 5.500000 | 5.5 | 5/6 tie | 1 | NA | NA | NA | flag | NA | NA | NA | NA | NA | NA |
| 06012CSR_Bug_2006-09-05_0 | BCG_PugLowWilVal_500ct | Hi | 0 | 0.0000000 | 0.0000000 | 0.7385321 | 0.2614679 | 0.0 | 1 | PASS | 4 | 0.7385321 | 5 | 0.2614679 | 0.4770642 | NA | 4.261468 | 4 | 4- | 0 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
| flag | NA | |
|---|---|---|
| brackish organisms present | 7 | 671 |
| catchment, Large | 0 | 678 |
| catchment, small | 158 | 520 |
| individuals, dominant 02, Large | 204 | 474 |
| individuals, Large | 0 | 678 |
| individuals, small | 80 | 598 |
| Low density (ft2) | 0 | 678 |
| Low density (m2) | 0 | 678 |
| Ramellogammarus | 8 | 670 |
| surface area, small | 112 | 566 |