Out-of-sample validation for structured expert judgment of Asian carp establishment in Lake Erie

Abstract

Structured expert judgment (SEJ) is used to quantify the uncertainty of nonindigenous fish (bighead carp [Hypophthalmichthys nobilis] and silver carp [H. molitrix]) establishment in Lake Erie. The classical model for structured expert judgment model is applied. Forming a weighted combination (called a decision maker) of experts' distributions, with weights derived from performance on a set of calibration variables from the experts' field, exhibits greater statistical accuracy and greater informativeness than simple averaging with equal weights. New methods of cross validation are applied and suggest that performance characteristics relative to equal weighting could be predicted with a small number (1-2) of calibration variables. The performance-based decision maker is somewhat degraded on out-of-sample prediction, but remained superior to the equal weight decision maker in terms of statistical accuracy and informativeness. Key Points: Structured expert judgment is applied to quantify uncertainty in Asian carp establishment in Lake Erie Calibration variables from the experts' field are used to enable performance-based combination In sample validation is applied to the combination of expert judgments and to the experts Out-of-sample validation is applied to the performance-based combination method