The Unintended Harm of Artificial Intelligence (AI): Exploring Critical Incidents of AI in Healthcare

Abstract

Artificial intelligence (AI) has been utilized in healthcare for years, presenting various risks. However, there is a gap in understanding AI incidents, particularly their impacts and associated risks. This study provides an overview of AI-related incidents in healthcare, focusing on the technologies involved, their contexts (e.g., medical conditions or treatments), and the effects of these incidents. We explored cases from the AI Algorithmic and Automation Incidents and Controversies (AIAAIC) repository, identifying 82 health-related incidents, including 14 cases with high and 1 case with critical severity. Our findings highlight significant concerns about AI misbehavior affecting patient safety, especially in medical monitoring, prediction, and diagnosis systems, including those not specifically designed for these functions (e.g., ChatGPT). The severity of these incidents underscores the need to enhance the robustness and reliability of AI systems, impacting trust in such technologies.