Validity of surgical simulation for the assessment of operative skill

Abstract

Background: Surgical simulators are being promoted as a means of assessing a surgeon's technical skills. Little evidence exists that simulator performance correlates with actual technical ability. This study was undertaken to determine the criterion and construct validity of currently available surgical simulations in the evaluation of technical skill. Methods: Simulator assessment was carried out on 36 basic surgical trainees, 37 surgically naïve first-year medical students and 16 experienced general surgical consultants. Some 26 trainees and 36 students underwent repeat assessment after 6 months. A previously validated, 19-point technical skill assessment form, based on direct observation of trainee performance in the operating theatre, was also completed by each trainee's supervising consultant. Results: An insignificant or weak correlation was found between simulator performance and both duration of basic surgical experience and consultant assessment of technical skill. Six months of basic surgical training led to an improvement in performance, not seen in an untrained control group, in only one of the six simulations tested. Discrimination between surgically naïve and experienced subjects was only demonstrated, in part, for four of the six tasks. Conclusion: The assessment of technical skill needs to be improved. Work is needed to establish the reliability and validity of currently available simulation models before they are formally introduced for high-stakes assessment.