On the use of the extreme dependency score to investigate the performance of an NWP model for rare events

Abstract

The extreme dependency score (EDS) has been recently proposed as a non-vanishing measure of skill to verify predictions of rare events. An idealized example is employed to investigate the properties of the EDS when the probability of the event happening is small, as is the case for rare events. The paper warns about the non-dependency of EDS on the false alarms and the correct rejections when the sample size is fixed, which encourages hedging. Thus, the EDS should not be the sole score used to assess a model's performance on forecasting rare events. The dependency of the EDS on the event probability (base rate) is also analysed. It is shown that the EDS can be written as a function of the base rate and the hit rate, and that only in a very particular condition the score is independent of the event probability. Therefore, when comparing forecasts skills in terms of the EDS, special attention has to be paid to separate the variations in skill from the variations of the probability that the event happens over time. The score is used to assess the performance of the ECMWF deterministic precipitation forecast over Europe for different forecast ranges and precipitation thresholds. Verification of precipitation forecasts in terms of EDS, combined with an assessment of the false alarm rate, shows a clear improvement of the ECMWF deterministic forecast for precipitation of rare events (i.e. large rainfall amounts). © 2009 Royal Meteorological Society.