Using qualitative data from a program director's evaluation form as an outcome measurement for medical school

Abstract

Background: Medical education programs need outcome measurements to promote curriculum improvement and to help meet accreditation standards. Purpose: Determine the added value of qualitative comments written by program directors (PDs) in response to a survey concerning fi rst postgraduate year (PGY-1) graduates. We hypothesized that these comments would serve as an additional outcome measurement for our graduates, adding information not readily captured in numeric data. Methods: PD evaluation form surveys from 1993-2002. All qualitative comments offered in response to free text questions were coded and compared with numeric ratings. Results: A total of 1,247 surveys were included (80% response rate). Comments about specifi c graduates were coded as positive, negative, or neutral and were categorized into themes. Inter-rater reliability was high ( κ = 0.82). Compared with 4% of graduates who received one or more numeric ratings of less than satisfactory, 7% had one or more qualitative phrases classifi ed as negative. Conclusions: Qualitative comments can serve as a useful outcome measurement.