Assessing the significance of model selection in ecology

Abstract

Model Selection is a key part of many ecological studies, with Akaike's Information Criterion (AIC) being by far the most commonly used technique for this purpose. Typically, a number of candidate models are defined a priori and ranked according to their expected out-of-sample performance. Model selection, however, only assesses the relative performance of the models and, as pointed out in a recent paper, a large proportion of ecology papers that use model selection do not assess the absolute fit of the 'best' model. In this paper, it is argued that assessing the absolute fit of the 'best' model alone does not go far enough. This is because a model that appears to perform well under model selection is also likely to appear to perform well under measures of absolute fit, even when there is no predictive value. This paper proposes a model selection permutation test that assesses the probability that the model selection statistic of the 'best' model could have occurred by chance alone, whilst taking account of dependencies between the models. It is argued that this test should always be performed as a part of formal model selection. The test is demonstrated on two real population modelling examples of ibex in northern Italy and wild reindeer in Norway. In both cases, the model selection permutation test gives a highly significant result, indicating that the performance of the 'best' model is unlikely to be through chance alone. R code is provided with which to perform the tests.