Controlling safety of artificial intelligence‐based systems in healthcare

Abstract

In response to the need to address the safety challenges in the use of artificial intelligence (AI), this research aimed to develop a framework for a safety controlling system (SCS) to address the AI black‐box mystery in the healthcare industry. The main objective was to propose safety guidelines for implementing AI black‐box models to reduce the risk of potential healthcare‐related incidents and accidents. The system was developed by adopting the multi‐attribute value model approach (MAVT), which comprises four symmetrical parts: extracting attributes, generating weights for the attributes, developing a rating scale, and finalizing the system. On the basis of the MAVT approach, three layers of attributes were created. The first level contained six key dimensions, the second level included 14 attributes, and the third level comprised 78 attributes. The key first level dimensions of the SCS included safety policies, incentives for clinicians, clinician and patient training, communication and interaction, planning of actions, and control of such actions. The proposed system may provide a basis for detecting AI utilization risks, preventing incidents from occurring, and developing emergency plans for AI‐related risks. This approach could also guide and control the implementation of AI systems in the healthcare industry.