Meaningful assessment of surgical expertise: Semantic labeling with data and crowds

Abstract

Many surgical assessment metrics have been developed to identify and rank surgical expertise; however,some of these metrics (e.g.,economy of motion) can be difficult to understand and do not coach the user on how to modify behavior. We aim to standardize assessment language by identifying key semantic labels for expertise. We chose six pairs of contrasting adjectives and associated a metric with each pair (e.g.,fluid/viscous correlated to variability in angular velocity). In a user study,we measured quantitative data (e.g.,limb accelerations,skin conductivity,and muscle activity),for subjects (n=3,novice to expert) performing tasks on a robotic surgical simulator. Task and posture videos were recorded for each repetition and crowd-workers labeled the videos by selecting one word from each pair. The expert was assigned more positive words and also had better quantitative metrics for the majority of the chosen word pairs,showing feasibility for automated coaching.