Verification and value of the Australian Bureau of Meteorology township seasonal rainfall forecasts in Australia, 1997-2005

Abstract

We verified the Bureau of Meteorology's seasonal rainfall forecasts for 262 townships throughout Australia, from its inception in June 1997 to May 2005. The results indicate that the forecasting system had low skill. Brier Skill Score and the receiver operating characteristic values were uniformly close to the no skill value. Forecast variances were consistently small. The overall observed variance was 0.0048, 2.1% of the variance of a perfect system. The estimate of the gradient of the outcome against forecast was 0.42 and was imprecise. Definitive statements about bias cannot be made. The value of the forecasts for decision-makers was estimated using value score curves, calculated for six forecast scenarios. All curves indicated that no economic benefit could have been reliably derived by users of the seasonal rainfall forecasts, with the exception of users with decisions triggered by a small shift in the forecast from climatology, in which case small economic gains may have occurred. Small value scores were associated with the observed forecast variance, not the observed bias. We examined the expected change in value scores associated with any future increase in forecast variance. This showed that a moderate increase from the observed variance would bring limited benefits. Substantial value to a broad range of users will only occur with a large increase in forecast variance. To deliver this, new lead indicators with markedly better predictive characteristics may need to be developed for the seasonal rainfall forecasting system.