Enabling Multi-modal Conversational Interface for Clinical Imaging

Abstract

Human-computer interaction research has to play a vital role in increasing the adoption of deep learning models in clinical settings, as their adoption is low despite models surpassing/matching the clinician's performance on many medical imaging tasks. Conversational AI has been successful as an interface for general information; however, there is a research gap for multi-modal conversational interface design for safety-critical clinical imaging systems. Our research points to the important role of multi-modal chat in improving usability and explainability through textual and visual explanations. Our main contributions include design principles for conversational interfaces in clinical imaging systems, the importance of multi-modal responses, and an understanding of the usefulness of mimicking clinician/radiologist interactions to improve usability. We show that diagnosis descriptions and visual responses improve the multi-modal conversational interface. The multi-modal conversational interface can help improve the adoption of deep learning systems in clinical settings, improving clinicians' efficiency and patient outcomes.