Deciphering ecology from statistical artefacts: Competing influence of sample size, prevalence and habitat specialization on species distribution models and how small evaluation datasets can inflate metrics of performance

Abstract

Aim: Sample size and species characteristics, including prevalence and habitat specialization, can influence the predictive performance of species distribution models (SDMs). There is little agreement, however, on which metric of model performance to use. Here, we directly compare AUC and partial ROC as metrics of SDM performance through analyses on the effects of species traits and sample size on SDM performance. Location: Three counties dominated by agricultural lands and coniferous forest in Oregon's Willamette Valley and Coast Range ecoregions. Methods: We systematically reduced a large avian point count dataset to alter sample sizes of 22 species of songbird. We used boosted regression trees to run SDMs for each species, quantified habitat specialization, and used mixed effects models to compare the influence of sample size, prevalence, and habitat specialization on SDM performance, calculated as AUC and partial ROC, across species. We calculated AUC and partial ROC with subset and independent evaluation data separately to more comprehensively investigate differences in metrics. Results: We found a positive quadratic effect of sample size and a strongly positive effect of habitat specialization on both metrics of model performance. We found a weak effect of prevalence on partial ROC and no effect in AUC. Contrary to expectations, when evaluated with a subset evaluation data, partial ROC was consistently highest in models with the smallest sample sizes. These small sample size models had correspondingly small sample sizes in subset evaluation datasets. Partial ROC evaluated with independent data and AUC evaluated with subset or independent data showed the expected positive correlation between sample size and model performance. Main Conclusions: We found that small evaluation datasets can artificially inflate partial ROC. With literature recommended minimum SDM sample sizes as low as three, attention must be given to the effects of correspondingly low sample sizes in evaluation datasets.