A degeneracy in cross-validated skill in regression-based forecasts

Abstract

Highly negative skill scores may occur in regression-based experimental forecast trials in which the data being forecast are withheld in turn from a fixed sample, and the remaining data are used to develop regression relationships - that is, exhaustive cross-validation methods. A small negative bias in skill is amplified when forecasts are verified using the correlation between forecasts and actual data. The same outcome occurs when forecasts are amplitude-inflated in conversion to a categorical system and scored in a "number of hits' framework. The effect becomes severe when predictor-predictand relationships are weak, as is often the case in climate prediction. Some basic characteristics of this degeneracy are explored for regression-based cross-validation. When the correlation skill score degeneracy is acknowledged and treated appropriately, cross-validation remains an effective and valid technique for estimating predictive skill for independent data. -from Authors