Evaluation of a structured application assessment instrument for assessing applications to Canadian postgraduate training programs in emergency medicine

Abstract

Objective: To determine the interrater reliability and predictive validity of a structured instrument for assessing applications submitted to a Fellow of the Royal College of Physicians of Canada (FRCP) emergency medicine residency program. Methods: An application assessment instrument was derived based on faculty and resident input, institutional and national documents, and previous protocols. The instrument provided a score based on objective anchors for each of four application components. Three assessors were introduced to the instrument in a detailed tutorial session. Assessors were given five applications to score and results were compared for understanding of the scoring principles. The instrument was used in a developmental pilot to assess the 2001 cohort of applications and revised again. Applications for the 2002 study cohort were submitted through a central application service. Assessors used the instrument to score each application independently. Interrater reliability was determined by calculating a two-way mixed-effect Cronbach's alpha. Results: Forty applications were received for the year 2002. Thirty-eight application packages were complete and data collection was complete for all 38. The single-rater reliabilities for the curriculum vitae, personal letter, transcript, reference letters, and overall package were 0.73, 0.52, 0.64, 0.61, and 0.72, respectively. The three-rater reliabilities for the components were 0.89, 0.77, 0.84, and 0.82, respectively. The three-rater reliability of the overall application score was 0.89. Conclusions: Three-rater reliabilities for each component and the entire application package were high. Multiple assessors are required to generate acceptable reliabilities. Using strict design and implementation principles can lead to a reliable instrument for assessing complex application packages.