Holistic and mechanical combination in psychological assessment: Why algorithms are underutilized and what is needed to increase their use

Abstract

Although mechanical combination results in more valid human performance predictions and decisions than holistic combination, existing publications suggest that mechanical combination is rarely used in practice. Yet, these publications are either descriptions of anecdotal experiences or outdated surveys. Therefore, in several Western countries, we conducted two surveys (total N = 323) and two focus groups to investigate (1) how decision makers in psychological assessment and human resource practice combine information, (2) why they do (not) use mechanical combination, and (3) what may be needed to increase its use in practice. Many participants reported mostly using holistic combination, usually in teams. The most common reasons for not using mechanical combination were that algorithms are unavailable in practice and that stakeholders do not accept their use. Furthermore, decision makers do not quantify information, do not believe in research findings on evidence-based decision making, and think that combining holistic and mechanical combination results in the best decisions. The most important reason why mechanical combination is used was to increase predictive validity. To stimulate the use of mechanical combination in practice, decision makers indicated that they should receive more training on evidence-based decision making and that decision aids supporting the use of mechanical combination should be developed.