Evaluation of Probabilistic Forecasts of Binary Events with the Neighborhood Brier Divergence Skill Score

Abstract

A procedure for evaluating the quality of probabilistic forecasts of binary events has been developed. This is based on a two-step procedure: pooling of forecasts on the one hand and observations on the other hand, on all the points of a neighborhood in order to obtain frequencies at the neighborhood length scale and then to calculate the Brier divergence for these neighborhood frequencies. This score allows the comparison of a probabilistic forecast and observations at the neighborhood length scale, and therefore, the rewarding of event forecasts shifted from the location of the observed event by a distance smaller than the neighborhood size. A new decomposition of this score generalizes that of the Brier score and allows the separation of the generalized resolution, reliability, and uncertainty terms. The neighborhood Brier divergence skill score (BDnSS) measures the performance of the probabilistic forecast against the sample climatology. BDnSS and its decomposition have been used for idealized and real cases in order to show the utility of neighborhoods when comparing at different scales the performances of ensemble forecasts between themselves or with deterministic forecasts or of deterministic forecasts between themselves. SIGNIFICANCE STATEMENT: A pooling of forecasts on the one hand and observations on the other hand, on all the points of a neighborhood, is performed in order to obtain frequencies at the neighborhood scale. The Brier divergence is then calculated for these neighborhood frequencies to compare a probabilistic forecast and observations at the neighborhood scale. A new decomposition of this score generalizes that of the Brier score and allows the separation of the generalized resolution, reliability, and uncertainty terms. This uncertainty term is used to define the neighborhood Brier divergence skill score which is an alternative to the popular fractions skill score, with a more appropriate denominator.