Effects of online frame-of-reference training on assessment accuracy in the objective structured clinical examination for physical therapy students

Abstract

Objectives: This study investigates how online frame-of-reference (FOR) training of raters of the objective structured clinical examination (OSCE) for physical therapy students affects assessment accuracy. Methods: The research was conducted in a 1-month-long randomized controlled trial. Participants: The participants were 36 physical therapists without experience assessing clinical skills using the OSCE. The training group completed the FOR training online, which was conducted once a week in two 90-minute sessions. The control group self-studied the rubric rating chart used in the assessment. As a measure of accuracy, weighted kappa coefficients were used to check the agreement between correct score and those assessment by the participant in the OSCE. Results: The scores of the training group were higher than those of the control group in both post-and follow-up assessments, showing significant differences. No significant difference was found based on the assessment time and group for the high-agreement groups. Furthermore, scores of the low-accuracy training group were higher in the post-and follow-up assessments than those in the pre-assessment, showing significant differences. Conclusions: Online FOR training of the raters of the OSCE for physical therapists improved the assessment accuracy of the raters who had low accuracy in the pre-assessment; this improvement was maintained.