Psychometric properties of a peer-assessment program to assess continuing competence in physical therapy

Abstract

Background. The College of Physiotherapists of Ontario implemented an Onsite Assessment to evaluate the continuing competence of physical therapists. Objective. This study was undertaken to examine the reliability of the various tools used in the Onsite Assessment and to consider the relationship between the final decision and demographic factors. Design. This was a psychometric study. Methods. Trained peer assessors (n=63) visited randomly selected physical therapists (n=106) in their workplace. Fifty-three physical therapists were exarnined by 2 assessors simultaneously. The assessment included a review of practice issues, record keeping, billing practices, the physical therapist's professional portfolio, and a chart-stimulated recall process. The Quality Management Committee made the final decision regarding the physical therapist's performance using the assessor's summary report. Generalizability theory was used to examine the interrater reliability of the tools. Correlation coefficients and regression analyses were used to exarnine the relationships between demographic factors and performance. Results. The majority of the physical therapists (88%) completed the program successfully, 11% required remediation, and 1% required further assessment. The interrater reliability of the components was above .70 for 2 raters' evaluations, with the exception of billing practices. There was no relationship between the final decision and age or years since graduation (r