Introducing a novel applicant ranking tool to predict future resident performance: A pilot study

Abstract

The purposes of this study are to (1) introduce our novel Applicant Ranking Tool that aligns with the Accreditation Council for Graduate Medical Education competencies and (2) share our preliminary results comparing applicant rank to current performance. After a thorough literature review and multiple roundtable discussions, an Applicant Ranking Tool was created. Feasibility, satisfaction, and critiques were discussed via open feedback session. Inter-rater reliability was assessed using weighted kappa statistic (κ) and Kendall coefficient of concordance (W). Fisher’s exact tests evaluated the ability of the tool to stratify performance into the top or bottom half of their class. Internal medicine and anesthesiology residents served as the pilot cohorts. The tool was considered user-friendly for both data input and analysis. Inter-rater reliability was strongest with intradisciplinary evaluation (W = 0.8–0.975). Resident performance was successfully stratified into those functioning in the upper vs. lower half of their class within the Clinical Anesthesia-3 grouping (p = 0.008). This novel Applicant Ranking Tool lends support for the use of both cognitive and noncognitive traits in predicting resident performance. While the ability of this instrument to accurately predict future resident performance will take years to answer, this pilot study suggests the instrument is worthy of ongoing investigation.