Evaluating presence-absence models in ecology: the need to account for prevalence

@article{citeulike:263947, abstract = { Summary1.Models for predicting the distribution of organisms from environmental data are widespread in ecology and conservation biology. Their performance is invariably evaluated from the percentage success at predicting occurrence at test locations.2.Using logistic regression with real data from 34 families of aquatic invertebrates in 180 Himalayan streams, we illustrate how this widespread measure of predictive accuracy is affected systematically by the prevalence (i.e. the frequency of occurrence) of the target organism. Many evaluations of presence\&\#150;absence models by ecologists are inherently misleading.3.With the same invertebrate models, we examined alternative performance measures used in remote sensing and medical diagnostics. We particularly explored receiver-operating characteristic (ROC) plots, from which were derived (i) the area under each curve (AUC), considered an effective indicator of model performance independent of the threshold probability at which the presence of the target organism is accepted, and (ii) optimized probability thresholds that maximize the percentage of true absences and presences that are correctly identified. We also evaluated Cohen\&\#146;s kappa, a measure of the proportion of all possible cases of presence or absence that are predicted correctly after accounting for chance effects.4.AUC measures from ROC plots were independent of prevalence, but highly significantly correlated with the much more easily computed kappa. Moreover, when applied in predictive mode to test data, models with thresholds optimized by ROC erroneously overestimated true occurrence among scarcer organisms, often those of greatest conservation interest. We advocate caution in using ROC methods to optimize thresholds required for real prediction.5.Our strongest recommendation is that ecologists reduce their reliance on prediction success as a performance measure in presence\&\#150;absence modelling. Cohen\&\#146;s kappa provides a simple, effective, standardized and appropriate statistic for evaluating or comparing presence\&\#150;absence models, even those based on different statistical algorithms. None of the performance measures we examined tests the statistical significance of predictive accuracy, and we identify this as a priority area for research and development.}, author = {Manel, S. and Williams, H. C. and Ormerod, S. J. }, citeulike-article-id = {263947}, issn = {0021-8901}, journal = {Journal of Applied Ecology}, keywords = {presence-absence-models, prevalence, roc}, month = {October}, number = {5}, pages = {921--931}, posted-at = {2005-07-24 21:03:27}, priority = {2}, title = {Evaluating presence-absence models in ecology: the need to account for prevalence}, url = {http://www.ingentaconnect.com/content/bsc/jappl/2001/00000038/00000005/art00647}, volume = {38}, year = {2001} }

TY - JOUR ID - citeulike:263947 L3 - citeulike-article-id:263947 TI - Evaluating presence-absence models in ecology: the need to account for prevalence JF - Journal of Applied Ecology VL - 38 IS - 5 SP - 921 EP - 931 SN - 0021-8901 N2 - Summary1.Models for predicting the distribution of organisms from environmental data are widespread in ecology and conservation biology. Their performance is invariably evaluated from the percentage success at predicting occurrence at test locations.2.Using logistic regression with real data from 34 families of aquatic invertebrates in 180 Himalayan streams, we illustrate how this widespread measure of predictive accuracy is affected systematically by the prevalence (i.e. the frequency of occurrence) of the target organism. Many evaluations of presence–absence models by ecologists are inherently misleading.3.With the same invertebrate models, we examined alternative performance measures used in remote sensing and medical diagnostics. We particularly explored receiver-operating characteristic (ROC) plots, from which were derived (i) the area under each curve (AUC), considered an effective indicator of model performance independent of the threshold probability at which the presence of the target organism is accepted, and (ii) optimized probability thresholds that maximize the percentage of true absences and presences that are correctly identified. We also evaluated Cohen’s kappa, a measure of the proportion of all possible cases of presence or absence that are predicted correctly after accounting for chance effects.4.AUC measures from ROC plots were independent of prevalence, but highly significantly correlated with the much more easily computed kappa. Moreover, when applied in predictive mode to test data, models with thresholds optimized by ROC erroneously overestimated true occurrence among scarcer organisms, often those of greatest conservation interest. We advocate caution in using ROC methods to optimize thresholds required for real prediction.5.Our strongest recommendation is that ecologists reduce their reliance on prediction success as a performance measure in presence–absence modelling. Cohen’s kappa provides a simple, effective, standardized and appropriate statistic for evaluating or comparing presence–absence models, even those based on different statistical algorithms. None of the performance measures we examined tests the statistical significance of predictive accuracy, and we identify this as a priority area for research and development. KW - presence-absence-models KW - prevalence KW - roc AU - Manel, S AU - Williams, HC AU - Ormerod, SJ PY - 2001/10// UR - http://www.ingentaconnect.com/content/bsc/jappl/2001/00000038/00000005/art00647 ER -