Same Same yet Different: Equivalence of the Multidimensional Assessment of Vocational Competence Across Test Formats

Abstract

The rapid digital transformation of vocational education and training (VET) has underscored the need to adapt traditional assessment methods to digital formats. However, when transitioning to digital modes, it is crucial to consider factors beyond mere technical implementation, particularly the potential impact of altered presentation formats on the validity of assessment outcomes. This study explores these considerations in the context of computer-based assessments, focusing on validating a digital format for evaluating vocational-economic competence. A particular emphasis was placed on ensuring measurement invariance between digital and paper-pencil formats, taking into account the effects of assessment layout, including cognitive load, the split-attention effect, and environmental factors such as context dependence and the digital literacy of test-takers. Furthermore, this study outlines a guideline-compliant validation procedure for the digital version of a vocational-economic competence assessment. To achieve this, we employed multi-group confirmatory factor analysis (MGCFA) and a multidimensional item response theory (IRT) model to assess measurement invariance and estimate person abilities across the two formats. Validity evidence obtained includes both measurement invariance across subgroups and the assessment's dimensionality.