Effective Continuous Quantitative Measures for End-to-End AI Guardrails

Abstract

Large Language Models such as ChatGPT have brought cutting-edge AI systems into the cultural zeitgeist. As a result, AI is no longer an isolated fief of academia or forward-leaning businesses. There are more than 35 million visits to open-source models in public repositories monthly. Clearly, the general technology community has caught onto the power of such systems and is keen to harness the promise of efficiency, productivity, and enhanced capability. Concurrent to this uptrend, AI systems are understood to be potentially vulnerable to various ethical issues. Such issues range from bias and fairness, to explainability and trustworthiness. More than mere theory, such vulnerabilities have manifested in mainstream settings such as politics, medicine, and law. The ethical implementation and operation of AI systems is, therefore, of critical interest as the democratization of such systems gains accelerates. However, there is an ongoing challenge insofar as there is little consensus on what constitutes quantitative ethical and responsible AI guardrails. This leaves AI practitioners without sufficient guidance to implement systems reasonably free from societal level harm. Accordingly, this work presents a structured taxonomy and concept matrix consisting of 39 discrete guardrails arrayed across a three-phased AI system lifecycle. Measure families further organize measures in terms such as bias mitigation, adversarial robustness, and anomaly monitoring. Then, I provide specific quantitative metrics for each measure construct. The intended takeaway is for AI practitioners to have the means to select appropriate and effective metrics for assuring ethical and responsible guardrails.