Measuring Professional Competence Using Computer-Generated Log Data

Abstract

One of the benefits of computer-based assessments lies in the automatic generation of log data. Such behavioural process data provide a time-stamped documentation of students’ interactions with the assessment system (e.g., mouse clicks). This chapter explores the usefulness of computer-generated log data for the measurement of professional competence and their potential for the research on professional learning and development. Based on a selection of studies, we illustrate how interindividual differences in task completion processes can be analysed with the help of log data, e.g. to identify the use of certain problem-solving strategies, or to reveal subgroups of students with efficiency barriers. We further present our own research, where we applied a theory on the diagnostic process (Abele, Vocat Learn 11(1):133–159, 2018) in order to assess diagnostic strategies (Abele and von Davier, CDMs in vocational education: assessment and usage of diagnostic problem-solving strategies in car mechatronics. In: von Davier M, Lee YS (eds) Handbook of diagnostic classification models. Springer International Publishing, pp 461–488. https://doi.org/10.1007/978-3-030-05584-4_22, 2019) in the domain of car mechatronics using log data. A profound understanding of interindividual process differences may supplement a merely product-oriented competence measurement and pave the way for a more process-oriented approach. Challenges concerning the assessment, analysis and interpretation of log data will be discussed.