Verification and Validation of AI Systems Using Explanations

Abstract

Verification and validation of AI systems, particularly learning-enabled systems, is hard because often they lack formal specifications and rely instead on incomplete data and human subjective feedback. Aligning the behavior of such systems with the intended objectives and values of human designers and stakeholders is very challenging, and deploying AI systems that are misaligned can be risky. We propose to use both existing and new forms of explanations to improve the verification and validation of AI systems. Toward that goal, we preseant a framework, the agent explains its behavior and a critic signals whether the explanation passes a test. In cases where the explanation fails, the agent offers alternative explanations to gather feedback, which is then used to improve the system's alignment. We discuss examples of this approach that proved to be effective, and how to extend the scope of explanations and minimize human effort involved in this process.