Reliability diagrams for multicategory probabilistic forecasts

Abstract

The most common method of verifying multicategory probabilistic forecasts such as are used in probabilistic quantitative precipitation forecasting is through the use of the ranked probability score. This single number description of forecast accuracy can never capture the multidimensional nature of forecast quality and does not inform the forecaster about the sources of forecast deficiencies. A new type of reliability diagram is developed here and applied to probabilistic quantitative precipitation forecasts from a university contest. This diagram is shown to potentially be useful in helping the forecaster to correct some errors in assigning the categorical probabilities.