How interpolation and resolution can affect verification scores: A study based on the Fractions Skill Score

Abstract

The Fractions Skill Score (FSS) has been used to routinely verify six-hourly UK precipitation forecasts since early 2008. For a 17-month period between May 2011 and September 2012 precipitation forecasts from the 4-km version of the Met Office Unified Model (UK4) were evaluated against two versions of radar-rainfall analyses, the 5-km product which had existed for many years, and a new 1-km product created using a revised algorithm. The standard practice is to interpolate the model forecast to the observation grid. This paper looks at the combined impact of the resolution of the verifying grid and interpolation on the magnitude of the FSS for six-hour accumulations. The 90th percentile threshold FSS mean scores based on the 1-km grid are higher for the 5-km neighbourhood. The 5- and 1-km verification grid scores are closest for the 25-km neighbourhood. For larger neighbourhoods there are several lead times and times of the day where the 5-km grid mean score is higher. In a cumulative sense the 1-km grid 90th percentile threshold FSS is 10-15 % higher for the 5-km neighbourhood. This is relative to the 5-km grid score and based on a 180-day running mean. Individual score differences are significant at the 5 % level. It suggests that the same 4-km model forecasts downscaled to a 1-km grid, and evaluated over a 5-km neighbourhood, score better than when upscaled (slightly) to a 5-km grid, which is close to the model grid scale. For the 25-km neighbourhood the cumulative relative score difference is reduced to less than ±3 % in the 180-day running means, though the differences for some lead times and times of day are significant at the 10 % level. The same trend is present for 51 and 101-km neighbourhoods, with an increasing number of lead times and times of day having a 5-km grid score which is higher. The choices made at the outset regarding interpolation and the detail (or lack thereof) in a verifying data set are influential enough to be non-neglible. For a spatial score such as the FSS the impact changes with neighbourhood size.