The computer-based assessment of domain-specific problem-solving competence—A three-step scoring procedure

Abstract

Problem-solving competence is an important requirement in back offices across different industries. Thus, the assessment of problem-solving competence has become an important issue in learning and instruction in vocational and professional contexts. We developed a computer-based low-stakes assessment of problem-solving competence in the domain of controlling. The assessment required participants to process three complex and authentic business-related scenarios within an office simulation. The low number of items and the open-ended response threaten the reliability of the scoring. We collected data from 780 German students in Vocational Education and Training (VET) and applied the following procedure to score the participants’ solutions to the three scenarios: A fine-grained scoring resulted in a comprehensive set of response patterns for each scenario. Experts assigned partial credit scores for each of these response patterns and received feedback based on point-biserial correlations of WLE scores for the partial credit items with the total score of the respective subdimension. Finally, a multi-dimensional model was calibrated based on these partial credits and this model was further developed. This procedure leads to satisfying EAP/PV reliabilities between.78 and.84 for the subdimensions and.89 for the overarching construct.