An extension of WeatherBench 2 to binary hydroclimatic forecasts

Abstract

Binary forecasts of hydroclimatic extremes play a critical part in disaster prevention and risk management. While the recent WeatherBench 2 provides a versatile framework for verifying deterministic and ensemble forecasts of continuous variables, this paper presents an extension to binary forecasts of the occurrence versus non-occurrence of hydroclimatic extremes. Specifically, 17 verification metrics of the accuracy and discrimination of binary forecasts are employed and scorecards are generated to showcase the predictive performance. A case study is devised for binary forecasts of wet and warm extremes obtained from both deterministic and ensemble forecasts generated by three data-driven models, i.e., Pangu-Weather, GraphCast and FuXi, and two numerical weather prediction products, i.e., the high-resolution forecasting (HRES) and ensemble forecasting (ENS) of the Integrated Forecast System (IFS) of the European Centre for Medium-Range Weather Forecasts (ECMWF). The results show that the receiver operating characteristic skill score (ROCSS) serves as a suitable metric due to its relative insensitivity to the rarity of hydroclimatic extremes. For wet extremes, the GraphCast tends to outperform the IFS HRES when using the total precipitation of ERA5 reanalysis data as the ground truth. For warm extremes, Pangu-Weather, GraphCast and FuXi tend to be more skillful than the IFS HRES within 3 d lead time but become less skillful as lead time increases. In the meantime, the IFS ENS tends to provide skillful forecasts of both wet and warm extremes at different lead times and at the global scale. Through diagnostic plots of forecast time series at selected grid cells, it is observed that at longer lead times, forecasts generated by data-driven models tend to be smoother and less skillful compared to those generated by physical models. Overall, the extension of WeatherBench 2 facilitates more comprehensive comparisons of hydroclimatic forecasts and provides useful information for forecast applications.