A Fallacy in the Use of Skill Scores

Abstract

Evidence is presented to show that the form of the Heidke skill score commonly used to evaluate forecast procedures is not reliable for the purpose for which it is intended. This is primarily because the standard against which the forecast accuracy is tested is not a true standard, since it is a function of the forecasts issued. Thus, even when applied to the same representative set of data, it is entirely possible for one skilled forecast technique to have a higher skill score but a lower forecast accuracy than a second skilled technique or an unskilled technique. A new skill score is proposed which retains the useful characteristics of the common skill score without its drawbacks. Finally, some brief comments are made regarding the weighting of data in a contingency table to obtain operationally more useful skill scores.