A multivariate generalizability analysis of data from a performance assessment of physicians' clinical skills

Abstract

Although multivariate generalizability theory was developed more than 30 years ago, little published research utilizing this framework exists and most of what does exist examines tests built from tables of specifications. In this context, it is assumed that the universe scores from levels of the fixed multivariate facet will be correlated, but the error terms will be uncorrelated because subscores result from mutually exclusive sets of test items. This paper reports on an application in which multiple subscores are derived from each task completed by the examinee. In this context, both universe scores and errors may be correlated across levels of the fixed multi-variate facet. The data described come from the United States Medical Licensing Examination® Step 2 Clinical Skills Examination. In this test, each examinee interacts with a series of standardized patients and each interaction results in four component scores. The paper focuses on the application of multivariate generalizability theory in this context and on the practical interpretation of the resulting estimated variance and covariance components.