Performance of the Standard Cabin Method

Comparison of BEAST Models and Error Rates to Detect Simulated Degradation from Multiple Data Sets

Stephanie A Strachan, Trefor Reynoldson

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

The objective of this study was to evaluate the performance of 3 bioassessment models for reference data sets collected from the Australia Capital Territory (ACT), the Yukon River Basin (YT), and the Laurentian Great Lakes (GL) built following the standard Canadian Aquatic Biomonitoring Network (CABIN) method. To evaluate the models, we used validation reference-site data, which were artificially impaired to simulate 3 levels of eutrophication by varying the proportions of sensitive, intermediate, and tolerant taxa. Models correctly classified 56 to 62% of reference sites. Type 1 errors (assessing reference sites as degraded) were high for all data sets and ranged from 30 to 75%, in part because the biological communities of the validation sites extended to or beyond the range of the reference-site data used to build the models. Capturing the full range of ecological variation with adequate sample size is critical for reference-condition approach (RCA)-type models. Type 2 errors (assessing degraded sites as in reference condition) varied greatly among data sets and for each reference group within each data set. Resource managers must carefully consider the risks associated with making errors. Thus, standard methods for quality assurance of assessment models should include simulated data so that error rates and adjusted assessment thresholds can be reported to ensure that degradation can be detected and that undisturbed sites are not mistakenly subjected to unnecessary management action.
Original languageEnglish
Pages (from-to)1225-1237
Number of pages13
JournalFreshwater Science
Volume33
Issue number4
DOIs
Publication statusPublished - 2014

Fingerprint

degradation
methodology
Australian Capital Territory
biological assessment
Yukon Territory
biomonitoring
Great Lakes
eutrophication
quality control
comparison
method
rate
managers
river basin
lake
resource
sampling

Cite this

@article{e5e121ef59104ce893b63a61022a67e6,
title = "Performance of the Standard Cabin Method: Comparison of BEAST Models and Error Rates to Detect Simulated Degradation from Multiple Data Sets",
abstract = "The objective of this study was to evaluate the performance of 3 bioassessment models for reference data sets collected from the Australia Capital Territory (ACT), the Yukon River Basin (YT), and the Laurentian Great Lakes (GL) built following the standard Canadian Aquatic Biomonitoring Network (CABIN) method. To evaluate the models, we used validation reference-site data, which were artificially impaired to simulate 3 levels of eutrophication by varying the proportions of sensitive, intermediate, and tolerant taxa. Models correctly classified 56 to 62{\%} of reference sites. Type 1 errors (assessing reference sites as degraded) were high for all data sets and ranged from 30 to 75{\%}, in part because the biological communities of the validation sites extended to or beyond the range of the reference-site data used to build the models. Capturing the full range of ecological variation with adequate sample size is critical for reference-condition approach (RCA)-type models. Type 2 errors (assessing degraded sites as in reference condition) varied greatly among data sets and for each reference group within each data set. Resource managers must carefully consider the risks associated with making errors. Thus, standard methods for quality assurance of assessment models should include simulated data so that error rates and adjusted assessment thresholds can be reported to ensure that degradation can be detected and that undisturbed sites are not mistakenly subjected to unnecessary management action.",
keywords = "BEAST, CABIN, Type 1 error, Type 2 error, bioassessment, predictive modeling, reference condition approach, simulated disturbance",
author = "Strachan, {Stephanie A} and Trefor Reynoldson",
year = "2014",
doi = "10.1086/678948",
language = "English",
volume = "33",
pages = "1225--1237",
journal = "Journal of the North American Benthological Society",
issn = "2161-9565",
publisher = "The Society for Freshwater Science",
number = "4",

}

Performance of the Standard Cabin Method : Comparison of BEAST Models and Error Rates to Detect Simulated Degradation from Multiple Data Sets. / Strachan, Stephanie A; Reynoldson, Trefor.

In: Freshwater Science, Vol. 33, No. 4, 2014, p. 1225-1237.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Performance of the Standard Cabin Method

T2 - Comparison of BEAST Models and Error Rates to Detect Simulated Degradation from Multiple Data Sets

AU - Strachan, Stephanie A

AU - Reynoldson, Trefor

PY - 2014

Y1 - 2014

N2 - The objective of this study was to evaluate the performance of 3 bioassessment models for reference data sets collected from the Australia Capital Territory (ACT), the Yukon River Basin (YT), and the Laurentian Great Lakes (GL) built following the standard Canadian Aquatic Biomonitoring Network (CABIN) method. To evaluate the models, we used validation reference-site data, which were artificially impaired to simulate 3 levels of eutrophication by varying the proportions of sensitive, intermediate, and tolerant taxa. Models correctly classified 56 to 62% of reference sites. Type 1 errors (assessing reference sites as degraded) were high for all data sets and ranged from 30 to 75%, in part because the biological communities of the validation sites extended to or beyond the range of the reference-site data used to build the models. Capturing the full range of ecological variation with adequate sample size is critical for reference-condition approach (RCA)-type models. Type 2 errors (assessing degraded sites as in reference condition) varied greatly among data sets and for each reference group within each data set. Resource managers must carefully consider the risks associated with making errors. Thus, standard methods for quality assurance of assessment models should include simulated data so that error rates and adjusted assessment thresholds can be reported to ensure that degradation can be detected and that undisturbed sites are not mistakenly subjected to unnecessary management action.

AB - The objective of this study was to evaluate the performance of 3 bioassessment models for reference data sets collected from the Australia Capital Territory (ACT), the Yukon River Basin (YT), and the Laurentian Great Lakes (GL) built following the standard Canadian Aquatic Biomonitoring Network (CABIN) method. To evaluate the models, we used validation reference-site data, which were artificially impaired to simulate 3 levels of eutrophication by varying the proportions of sensitive, intermediate, and tolerant taxa. Models correctly classified 56 to 62% of reference sites. Type 1 errors (assessing reference sites as degraded) were high for all data sets and ranged from 30 to 75%, in part because the biological communities of the validation sites extended to or beyond the range of the reference-site data used to build the models. Capturing the full range of ecological variation with adequate sample size is critical for reference-condition approach (RCA)-type models. Type 2 errors (assessing degraded sites as in reference condition) varied greatly among data sets and for each reference group within each data set. Resource managers must carefully consider the risks associated with making errors. Thus, standard methods for quality assurance of assessment models should include simulated data so that error rates and adjusted assessment thresholds can be reported to ensure that degradation can be detected and that undisturbed sites are not mistakenly subjected to unnecessary management action.

KW - BEAST

KW - CABIN

KW - Type 1 error

KW - Type 2 error

KW - bioassessment

KW - predictive modeling

KW - reference condition approach

KW - simulated disturbance

UR - http://www.scopus.com/inward/record.url?scp=84937577801&partnerID=8YFLogxK

UR - http://www.mendeley.com/research/performance-standard-cabin-method-comparison-beast-models-error-rates-detect-simulated-degradation-m

U2 - 10.1086/678948

DO - 10.1086/678948

M3 - Article

VL - 33

SP - 1225

EP - 1237

JO - Journal of the North American Benthological Society

JF - Journal of the North American Benthological Society

SN - 2161-9565

IS - 4

ER -