Validity of constructed-response situational judgement tests in health professions education: A systematic review and meta-analysis

Abstract

Introduction: Constructed-response situational judgement tests (CR-SJTs) are used internationally to assess personal and professional attributes in health professions admissions, with over one million applicants to more than 500 programs having used them in the last decade. Despite this, a synthesis of their validity is lacking. This study aimed to quantify the association between CR-SJT scores and measures of personal, interpersonal and professional performance in health professions education and to determine how moderating factors influence this relationship. Methods: MEDLINE, EMBASE, CINAHL, ERIC, SCOPUS, Web of Science and grey literature sources (ProQuest Dissertations & Theses, EThOS and OpenGrey) were searched from inception to 7 April 2025. Search was supplemented by contacting experts and admissions directors to include unpublished quality assurance studies. Eligible studies evaluated a CR-SJT for applicants or trainees in health professions programmes and reported a quantitative relationship with a non-academic outcome (e.g., professionalism and communication). Of 463 full-texts reviewed, 27 met inclusion criteria. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, two reviewers independently extracted data and assessed risk of bias using the Quality In Prognosis Studies (QUIPS) tool. A multilevel random-effects meta-analysis was used to pool Fisher z-transformed correlation coefficients. Meta-regressions tested moderator effects, and sensitivity analyses examined bias impact. The primary outcome was the correlation between CR-SJT scores and measures of interpersonal or professional skills. Moderators included the construct congruence between the CR-SJT and the outcome measure, outcome type, publication type, and outcome assessment stage. Results: The 27 studies yielded 100 unique effect sizes and were judged, in total, to be at moderate risk of bias. The pooled correlation between CR-SJT scores and outcomes was z = 0.22 (95% CI, 0.16–0.28; p < 0.001). Construct congruence was the only significant moderator; more congruent outcomes showed z = 0.32 (95% CI, 0.25–0.37), compared to that of less congruent outcomes (z = 0.17; 95% CI, 0.12–0.18). Publication bias was insignificant (Egger's test, p = 0.73). Discussion: The use of CR-SJTs in health professions selection is supported by the evidence. Validity depends substantially on construct congruence between the CR-SJT and the outcome measure. Programs should consider CR-SJTs within their operational context and with deliberate attention to downstream evaluation alignment.