Testing competing precipitation forecasts accurately and efficiently: The Spatial Prediction Comparison Test

Abstract

Which model is best? Many challenges exist when testing competing forecast models, especially for those with high spatial resolution. Spatial correlation, double penalties, and small-scale errors are just a few such challenges. Many new methods have been developed in recent decades to tackle these issues. The spatial prediction comparison test (SPCT), which was developed for general spatial fields and applied to wind speed, is applied here to precipitation fields; which pose many unique challenges in that they are not normally distributed, are marked by numerous zero-valued grid points, and verification results are particularly sensitive to small-scale errors and double penalties. The SPCT yields a statistical test that solves one important issue for verifying forecasts spatially by accounting for spatial correlation. Important for precipitation forecasts is that the test requires no distributional assumptions, is easy to perform, and can be applied efficiently to either gridded or nongridded spatial fields. The test compares loss functions between two competing forecasts, where any such function can be used, but most still suffer from the limitations of traditional gridpoint-bygridpoint assessment techniques. Therefore, two new loss functions to the SPCT are introduced here that address these concerns. The first is based on distance maps and the second on image warping. Results are consistent with other spatial assessment methods, but provide a relatively straightforward mechanism for comparing forecasts with a statistically powerful test. The SPCT combined with these loss functions provides a new mechanism for appropriately testing which of two competing precipitation models is best, and whether the result is statistically significant or not. © 2013 American Meteorological Society.