Effect of Holistic Review, Interview Blinding, and Structured Questions in Resident Selection: Can we Predict Who Will Do Well in a Residency Interview?

Abstract

Objective: To examine the Urology residency application process, particularly the interview. Historically, the residency interview has been vulnerable to bias and not determined to be a predictor of future residency performance. Our goal is to determine the relationship between pre-interview metrics and post-interview ranking using best practices for Urology resident selection including holistic review, blinded interviews, and structured behaviorally anchored questions. Methods: Applications were assessed on cognitive (Alpha Omega Alpha, class rank, junior year clinical clerkship grades) and non-cognitive attributes (letters of recommendation [LOR], personal statement [PS], demographics, research, personal characteristics) by reviewers blinded to USMLE scores and photograph. Interviewers were blinded to the application other than PS and LORs. Interviews consisted of a structured behaviorally anchored question (SBI) and an unstructured interview (UI). Odds ratios were determined comparing pre-interview and interview impressions. Results: Fifty-one applicants were included in the analysis. USMLE step 1 score (average 245) was associated with Alpha Omega Alpha, class rank, junior year clinical clerkship, and PS. The UI score was associated with the LOR (P = .04) whereas SBI scores were not (P = .5). Faculty rank was associated with SBI, UI, and overall interview (OI) scores (P < .001). Faculty rank was also associated with LOR. Resident impression of interviewees were associated with faculty interview scores (P = .001) and faculty rank (P < .001). Conclusion: Traditional interviews may be biased toward application materials and may be balanced with behavioral questions. While Step 1 score does not offer additional information over other PI metrics, blinded interviews may offer discriminant validity over a PI rubric.