Specialized foundation models for intelligent operating rooms

Abstract

Surgical procedures unfold in complex environments demanding coordination between surgical teams, tools, imaging and increasingly, intelligent robotic systems. While AI solutions like ChatGPT and Gemini have revolutionized language understanding and seen early adaptions in clinical diagnosis, they fall short in the safety-critical, multimodal setting of surgery. Ensuring safety and efficiency in ORs of the future requires intelligent systems, like surgical robots, smart instruments and digital copilots, capable of understanding complex activities and hazards. We introduce ORQA, a multimodal foundation model unifying visual, auditory, and structured data for holistic surgical understanding. ORQA’s question-answering framework empowers diverse tasks, serving as an intelligence core for surgical technologies. We benchmark ORQA against generalist vision-language models, and show that while they struggle to perceive surgical scenes, ORQA delivers substantially stronger, consistent performance. To meet diverse deployment needs, we design, and release a family of smaller ORQA models tailored to different computational requirements. This work establishes a foundation for the next wave of intelligent surgical solutions, enabling surgical teams and medical technology providers to create smarter and safer operating rooms.