Assessing the fidelity of predictability estimates

Abstract

Predictability is an intrinsic limit of the climate system due to uncertainty in initial conditions and the chaotic nature of the atmosphere. Estimates of predictability together with calculations of current prediction skill are used to define the gaps in our prediction capabilities, inform future model developments, and indicate to stakeholders the potential for making forecasts that can inform their decisions. The true predictability of the climate system is not known and must be estimated, typically using a perfect model estimate from an ensemble prediction system. However, different prediction systems can give different estimates of predictability. Can we determine which estimate of predictability is most representative of the true predictability of the climate system? We test three metrics as potential indicators of the fidelity of predictability estimates in an idealized framework—the spread-error relationship, autocorrelation and skill. Using the North American multi-model ensemble re-forecast database, we quantify whether these metrics accurately indicate a model’s ability to properly estimate predictability. It is found that none of these metrics is a robust measure for determining whether a predictability estimate is realistic for El Nino-Southern oscillation events. For temperature and precipitation over land, errors in the spread-error ratio are related to errors in estimating predictability at the shortest lead-times, while skill is not related to predictability errors. The relationship between errors in the autocorrelation and errors in estimating predictability varies by lead-time and region.