Overview of methods for the verification of quantitative precipitation forecasts

Abstract

In the area of hydrological risk management, both Quantitative Precipitation Estimates (QPE) and Quantitative Precipitation Forecasts short time scales, i.e., for relatively small river and urban catchments. In such a context, forecasting can be viewed as the attempt to reduce the uncertainty of the future state of the hydrometeorological system and so anticipate mitigating actions. Authorities, however, often are still reluctant to devise and invest in such actions based on forecasts when their quality is unknown. In other words, for forecasts to be useful and effective the forecast quality and forecast uncertainty must be quantified. Much effort has been and is being invested in the quest of working with imperfect precipitation observations and forecasts. A number of initiatives are underway, such as the Hydrologic Ensemble Prediction EXperiment (HEPEX, Schaake et al. 2007) which is an international project established by the hydrological and meteorological communities. The mission of HEPEX is to demonstrate how to produce reliable hydrological ensemble predictions that can be used with confidence by emergency management and water resources sectors to make decisions that have important consequences for economy, public health and safety. The COST 731 Action (Rossa et al. 2005) is a European initiative which deals with the quantification of forecast uncertainty in hydrometeorological forecast systems. It is linked to the MAP D-PHASE initiative (www.map.meteoswiss.ch), a WWRP Forecast Demonstration Project (FDP), which is to provide evidence of the progress meteorological and hydrological modeling has achieved over the last decade or so. A characteristic of an FDP is that strict evaluation protocols are established to demonstrate and document such progress. Indeed, many atmospheric and hydrological forecast systems participate in this effort. The atmospheric part includes nowcasting based on radar, very high resolution next-generation numerical weather prediction (NWP) models, operational models, as well as a number of limited area ensemble prediction systems. In all of this verification, and verification of precipitation forecasts in particular, is fundamental! It is safe to say that the more detailed the forecasts the more complex the corresponding verification task. For example, verification of geostrophic flow can be viewed as relatively simple when compared to verification of turbulent flow. Precipitation is a stochastic quantity and exhibits fractal properties down to very small scales (e.g., Zawadzki 1973). It is difficult to observe, simulate and to verify. Furthermore, many more efforts have been invested in the development of forecasting techniques than in verification methodologies. This may be connected to the fact that the traditional approaches to verification of gridded forecasts were developed on relatively low resolution global NWP models to check the consistency of upper air fields against model analyses. Stanski et al. (1989) provide a thorough compilation of the statistics involved in NWP verification, while Wilks (2006) is an excellent text and reference book for statistical methods in the atmospheric sciences, covering forecast verification. However, with increasing resolution of the limited area models, verification of weather elements against observations has become a more complex problem. For example, while for medium-range forecasting typically daily rainfall accumulations are verified, the higher resolution meso-scale models are expected to have skill also in shorter time scales. Their performance is tested for shorter accumulation periods where for instance the timing and location of a frontal passage is essential and the traditional verification methods are not necessarily sufficient. Small positioning errors in the forecasts may result in the socalled double penalty: the verification measure tends to penalize rather than reward the models capability to provide some sort of information on small scale features (see Sect. 16.2.2). These issues are accentuated when it comes to verifying high resolution QPFs. The necessity to evaluate and justify the advantages of the ever higher resolution over the computationally less expensive coarser resolution NWP in terms of QPF quality has stimulated radically different verification approaches for spatial forecast fields over the last decade or so. These methods go well beyond point-to-point pair verification and borrow ideas from fields such as image and signal processing. The main lines of extension to judge whether or not a precipitation forecast for a given time and location is correct is to ask the question whether the main characteristics of fields are captured in the simulation. In other words, conditions for right and wrong are relaxed from at a given point and time in several ways. For example, in the class of neighborhood methods the condition of correct location is successively relaxed to yield an effective scale-dependent measure of forecast goodness (Ebert 2008). Harris et al. (2001) investigate whether the characteristic scales of rainfall fields are successfully reproduced, without necessarily requiring correspondence in location, while Ebert and McBride (2000) look for corresponding rain objects and decompose the measure for quality in components for matching location, amount and structure. Davis et al. (2006) take the description of precipitation objects one step further but still require object matching between the forecast and the observations. For hydrological applications the localization of precipitation is important on the scale of the considered catchment, so that it is useful to perform QPF verification on river basin averages (e.g., Oberto et al. 2006). Wernli et al. (2008) combine the idea of verifying precipitation within a predefined area, say a medium to large river catchment, in which not just the average rainfall amount is evaluated but also the average capability of the model to predict location and structure of the rainfall field, measures that do not require object correspondence. Datasets on which these methodologies are applied can span several years in order to try to document improvements in forecast quality. Improvements have been reported for parameters like the pressure or the temperature, but not for QPF (Hense et al. 2003). Performing verification over a full year will effectively mix a number of different flow regimes which, in theory, can present different challenges to a modeling system. Also, the verification results can be biased towards the most frequent regime, e.g. days with no intense weather. It is, therefore, quite common practice to differentiate verification for the four seasons, while it is far less common to perform a systematic separation of distinct flow regimes in which a forecast system may have different challenges to get realistic QPF. The diversity of approach emerging from these examples, which are detailed further in Sects. 16.3 to 16.5, document the efforts of the scientific and operational community to find adequate measures to describe forecast quality of high resolution QPF. However, such a variety holds the risk that verification results become difficult to compare. There have been several efforts to harmonize verification activities in the recent past. ECMWF, for example, compiled a set of recommendations for their member states (Nurmi 2003), while the Joint Working Group on Verification (JWGV) provided a survey of verification methods of weather elements and severe weather events (Bougeault 2002) and recommendations for the verification and intercomparison of QPFs from operational NWP models (JWGV 2004). There is an ongoing exercise in which the more recent verification techniques are to be compared on a set of common cases (ICP 2007). Probabilistic QPF is a promising avenue of improvement for high resolution rainfall prediction (e.g., Mittermaier 2007). The main ideas behind probabilistic forecasting are based on the imperfect knowledge of initial conditions and key parameters in parameterization schemes of mainly moist processes. Ensemble forecasting, i.e., forecasts starting from slightly differing initial conditions, is an established technique for estimating forecast uncertainty of the global models in the medium range. It has become increasingly popular also for high resolution limited area models in shorter time ranges, as well as in nowcasting. The radar community has started recently to produce probabilistic QPEs based on the error characteristics of radar measurements (Germann et al. 2006). Probabilistic forecasting is adding considerable complexity to the verification problem in that right and wrong no longer have a strict sense when it comes to a single forecast observation pair. Verification needs to take the frequency of occurrence of events into account. These issues are, however, beyond the scope of this Chapter and will not be discussed.