A skill score based on economic value for probability forecasts

Abstract

An approach to evaluating probability forecasts for dichotomous events, based on their economic value over all possible cost/loss ratio decision problems, is proposed. The resulting Value Score (VS) curve shows non-dimensionalised relative economic value as a function of the cost/loss ratios for different decision-makers, over their full meaningful range. The VS curve is similar in terms of computational mechanics and graphical display to the Relative Operating Characteristic (ROC) curve, but the ROC curve is shown to be insensitive to either conditional or unconditional biases and thus to reflect potential rather than actual skill. The possibility of collapsing the VS curve into a single scalar score is addressed, and it is shown that the results can depend very strongly on the assumed distribution of cost/loss ratios in the community of forecast users.