Do Preceptors With More Rating Experience Provide More Reliable Assessments of Medical Student Performance?

Abstract

Background: Although the existing psychometric literature provides guidance on the best method for acquiring a reliable clinical evaluation form (CEF)-based score, it also shows that a single CEF rating has very low reliability. Purpose: This study examines whether experience with rating students might act as a form of rater training and hence improve the quality of CEF ratings. Methods: Preceptors were divided into two groups based on rater experience. The univariate and multivariate G study designs used were simple rater (r)-nested-within-person (p) [r: p and r ○:p •] models, and in the univariate analysis was applied separately to CEFs completed by high and low experienced raters. Results: The high experienced rater group yielded a substantially higher observed reliability in both the univariate and multivariate analyses. Conclusions: These results support the hypothesis that high experienced raters produce more reliable ratings of student performance and suggest methods for improving CEF ratings. © 2012 Taylor and Francis Group, LLC.