Validating expert judgment with the classical model

Abstract

The classical model derives performance based weights for combining expert judgments based on calibration or seed variables from the experts’ field. Since publication of the TU Delft expert judgment database in 2008, various authors have attempted to use this data base for cross validation, splitting the seed variables into training sets and test sets. These attempts are reviewed. Many pitfalls and biases in cross validation efforts are identified and explained. A proposal for performing cross validation, based largely on the work of Eggstaff et al. (Reliab Eng Syst Saf 121:72–82, 2014) is formulated and illustrated with data from recent expert judgment studies.