Reliability, Number of Stations, and Examination Length in an Objective Structured Clinical Examination

Abstract

Objective Structured Clinical Examinations (OSCEs) are widely used to document medical competence but reliability concerns raise questions about the number of stations and time that should be allocated for reproducible performance estimates. For example, when Standardized Patients (SPs) are used, Swanson and Norcini have estimated that 8 hours of resting or approximately 24 twenty-minute stations are needed to reliably assess information gathering and communication skills.(1) The University of Michigan's (UM) third-year OSCE consists of 12 stations using a variety of formats to assess a range of clinical skills. Generalizability theory was used to assess the reliability of the existing examination and estimate potential changes needed for improvement. The results indicated that the 12 station exam has a reliability of approximately .60. if the exam were expanded to 20 stations, reliability would increase to the lower .70s. In order to obtain score reliability of 0.80, it would be necessary to have approximately 30 stations or 10 hours. The fact that the UM OSCE was designed to assess a range of clinical skills may introduce variance from content area and skill domain as well from student and test-related sources and thereby account for the relatively lower reliability estimates compared with the Swanson and Norcini study.