Log-optimal economic evaluation of probability forecasts

Abstract

The commercial test of an expert's probability assessments is not that they are accurate in an abstract sense, but that they yield financial returns to decision makers. From this utilitarian standpoint, a model or forecaster is merely a font of cash pay-offs, like any other form of asset or security. The modern perspective in finance theory is that individual 'securities' (sources of cash pay-offs) must be valued in portfolio rather than of themselves. Applying portfolio methods to forecast evaluation, the theoretical worth of a forecast depends on its marginal contribution to the best available portfolio of securities. When considered within a log-optimal (maximum E[log(wealth)]) portfolio, the value of an individual forecast (or forecaster) depends on both its expected cash pay-off and the covariance of its pay-off with those from all other available securities. In effect, portfolio theory rewards forecasters more for making accurate forecasts when other forecasters (or, more broadly, other sources of pay-offs) perform badly than when all or most forecasters do well. Conversely, the penalty for being wrong is reduced when other forecasters are right. This has the effect of promoting original thinking and unique (idiosyncratic) forecasting expertise. Herding by resort to industry standard models or routines is discouraged. © 2012 Royal Statistical Society.