Consistency of interrater scoring of student performances of osteopathic manipulative treatment on COMLEX-USA level 2-PE

Abstract

Context: Assessment of osteopathic manipulative treatment (OMT) is included in the National Board of Osteopathic Medical Examiners' Comprehensive Osteopathic Medical Licensing Examination-USA Level 2-Performance Evaluation (COMLEX-USA Level 2-PE). The scores earned for OMT should be equivalent among all raters regardless of which technique is scored or which rater is scoring the performance. As a quality assurance measure, selected examination dates and the encounters within the administration of COMLEX-USA Level 2-PE are scored by 2 raters: first by a "live" rater and next by a quality assurance rater. Neither rater knows if he or she is the first or second rater. Objective: To compare candidate's scores recorded for OMT on COMLEX-USA Level 2-PE to determine whether differences exist among raters and techniques scored. Methods: The authors evaluated candidate performances that took place from July through November 2012. For each performance, 2 raters scored the same technique or different techniques using the OMT scoring rubric. Discrepancies between scores were compared using t tests. Statistical significance was set at P