A qualitative interview study investigating patient, health professional, and developer perspectives on real-world implementation of patient-centered AI systems

Abstract

Artificial intelligence (AI) systems in healthcare often fail to improve patient outcomes despite high development accuracy. We conducted semi-structured interviews with patients (n = 18), health professionals (n = 8), and AI developers (n = 8), using a postpartum depression risk algorithm as a use case. Through thematic analysis informed by sociotechnical frameworks, we identified six themes: harm mitigation, clinical utility, communication strategies, data quality, privacy/security, and responsible governance. All stakeholders emphasized that patient-centered AI must provide actionable benefits while minimizing bias, stigma, and anxiety. Patients wanted professional interpretation of AI outputs. Participants identified tensions between explainability and accuracy, varying patient preferences for accessing predictions, and unclear accountability when AI recommendations cause adverse outcomes. Our findings support patient-centered implementation through four strategies: providing professionals with competencies and protected time; engaging stakeholders throughout development; offering flexible communication accommodating diverse health literacy; and establishing multi-layered governance with shared accountability across developers, professionals, and institutions.