The Reliability of a Personal Statement as a Selection Tool in the Admissions Process

Abstract

The job of the Admissions Committee is to choose students who will meet the requirements of the profession. In an effort to improve the admissions process and include qualities other than high grades, the Department of Physical Therapy developed a Personal Statement: a 17-item questionnaire to which applicants must respond. Nine subscores are available for each Personal Statement. Personal Statements from the intake years of 1993 and 1994 were used in this study. Each Statement is read by three raters, who are provided with a detailed (8-page) scoring guideline. Generalizability analysis was carried out. The error variance comprised from 41 to 45% of the total variance. Subject by item variance was 27% to 35%, subject by rater variance was 8.5 to 9% and subject variance comprised 15 to 19% of the total variance. A large part of the variance remains unaccounted for. The Interrater generalizability coefficient was 0.47–0.5. Using the Spearman Brown formula, Interrater reliability using three raters was calculated to be 0.73–0.75. Item reliabilities were between 0.19 and 0.63, with one exception: subscore 7, the “problem-solving ” component of the Personal Statement, had an item reliability of 0.06–0.08. The Personal Statement, as a whole, has adequate interrater and item reliabilities, with the exception of the subscore related to problem-solving. Total test reliability was 0.76–0.78. Overall, there is reasonable Interrater reliability, good test reliability, and good item reliability, with the exception of the problem-solving component.