Situation Awareness Assessment for Anesthesia Residents (SAAAR): Development and Preliminary Evaluation of a Multimodal System

Abstract

Objective: This study reports the development and preliminary evaluation of SAAAR, a multimodal system designed to assess and support the development of situation awareness (SA). Background: SA is critical in anesthesiology, yet existing assessment methods lack standardized tools tailored to its complexities of anesthetic practice. Systems developed in other domains have limited applicability, highlighting the need for a purpose-built approach for anesthesia residents. Method: The SAAAR comprises two components: a 16-item behavioral marker scale and a structured debriefing with eye-tracking. Thirteen anesthesiology faculty tested interrater and test-retest reliability, while five experts conducted content validation of the scale. Both components were implemented in a simulation-based training program for preliminary system evaluation. Results: The behavioral marker scale demonstrated moderate content validity and high reliability. Internal consistency was strong (McDonald’s Ω = 0.928), test-retest reliability high (Spearman’s ρ = 0.952), and interrater agreement moderate (Kendall’s W = 0.412). Faculty reported the scale to be clear, comprehensive, and easy to use. Pilot implementation showed significant improvements across domains (Wilcoxon signed-rank test), indicating the system’s potential to provide targeted feedback and guide educational interventions. Conclusions: Grounded in HFE principles, the SAAAR provides a structured approach to assessing SA in anesthesia residents and demonstrates preliminary potential to inform educational strategies. Further research is required to determine its impact on clinical performance. Application: The SAAAR offers residency programs and human factors experts a practical tool for assessing SA and designing targeted training. Its adaptable framework suggests potential applicability in other high-pressure medical contexts, pending further evaluation.