NeuroLens: organ localization using natural language commands for anatomical recognition in surgical training

Abstract

Purpose: This study introduces NeuroLens, a multimodal system designed to enhance anatomical recognition by integrating video with textual and voice inputs. It aims to provide an interactive learning platform for surgical trainees. Methods: NeuroLens employs a multimodal deep learning localization model trained on an Endoscopic Third Ventriculostomy dataset. It processes neuroendoscopic videos with textual or voice descriptions to identify and localize anatomical structures, displaying them as labeled bounding boxes. Usability was evaluated through a questionnaire by five participants, including surgical students and practicing surgeons. The questionnaire included both quantitative and qualitative sections. The quantitative part covered the System Usability Scale (SUS) and assessments of system appearance, functionality, and overall usability, while the qualitative section gathered user feedback and improvement suggestions. The localization model’s performance was assessed using accuracy and mean Intersection over Union (mIoU) metrics. Results: The system demonstrates strong usability, with an average SUS score of 71.5, exceeding the threshold for acceptable usability. The localization achieves a predicted class accuracy of 100%, a localization accuracy of 79.69%, and a mIoU of 67.10%. Participant feedback highlights the intuitive design, organization, and responsiveness while suggesting enhancements like 3D visualization. Conclusion: NeuroLens integrates multimodal inputs for accurate anatomical detection and localization, addressing limitations of traditional training. Its strong usability and technical performance make it a valuable tool for enhancing anatomical learning in surgical training. While NeuroLens shows strong usability and performance, its small sample size limits generalizability. Further evaluation with more students and enhancements like 3D visualization will strengthen its effectiveness.