Toward Operational Safety Verification of AI-Enabled CPS

Abstract

AI-enabled Cyber-physical systems (CPS) such as artificial pancreas (AP) or autonomous cars are using machine learning to make several critical decisions. The system is subject to inputs and scenarios which are not observed during training and the expected outputs are not known. Hence, popular model based verification techniques that characterize behavior of a control system before deployment using predictive models may be inaccurate and often result in incorrect safety analysis results. In addition, regulatory agencies are required to regulate safety-critical AI enabled CPS to ensure their operational safety. However, high complexity of the system result in myriad of safety concerns all of which may not only be comprehensively tested before deployment but also may not even be detected during design and testing phase. In this work, we propose a tool to help regulatory agencies compare the operation of the CPS with the specifications given by the manufacturer to ensure that the operation results conform with the safety assured design of a CPS.