Reliability and validity of performance evaluations of pain medicine clinical faculty by residents and fellows using a supervision scale

Abstract

BACKGROUND: Annual and/or semiannual evaluations of pain medicine clinical faculty are mandatory for multiple organizations in the United States. We evaluated the validity and psychometric reliability of a modified version of de Oliveira Filho et al clinical supervision scale for this purpose. METHODS: Six years of weekly evaluations of pain medicine clinical faculty by resident physicians and pain medicine fellows were studied. A 1-4 rating (4 = "Always") was assigned to each of 9 items (eg, "The faculty discussed with me the management of patients before starting a procedure or new therapy and accepted my suggestions, when appropriate"). RESULTS: Cronbach α of the 9 items equaled.975 (95% confidence interval [CI], 0.974-0.976). A G coefficient of 0.90 would be expected with 18 raters; the N = 12 six-month periods had mean 18.8 ± 5.9 (standard deviation [SD]) unique raters in each period (median = 20). Concurrent validity was shown by Kendall τb= 0.45 (P <4. There were 3 of 13 ratees with significantly more averages <4 than the other ratees, based on P