Evaluating Modes of Observations Using Hierarchical Signal Detection Rater Models

Abstract

Scoring of performance can be based on different modes of observation, which can include live and videotaped observations. Although live observations have been the traditional format of measuring examinee performance in education and in healthcare, videotaped observations provide educators and testing agencies the promise of unbiased and standardized evaluations, offering practical solutions to limitations of real-time scoring. This study proposes a measurement model taking into account different modes of observation, using an extension of the hierarchical rater model based on signal detection theory (HRM-SDT). A hierarchical rater model is motivated by the nested structure of scores assigned by raters–scores assigned by raters become indicators of performance quality, which in turn become indicators of examinee ability. This study extends the hierarchical structure of the scoring process to include modes of observation, which serves as an intermediary level between the latent categorical indicator of performance quality and examinee ability, forming a three-level HRM-SDT. Analyses based on real-world data showed differences in the quality of scores from live observations and videotaped recordings. Compared to the traditional HRM-SDT, the overall model fit improved when including modes of observation. Simulations using different sample sizes and conditions provide implications for uses of this model.