A Framework to Monitor Machine Learning Systems Using Concept Drift Detection

Abstract

As more and more machine learning based systems are being deployed in industry, monitoring of these systems is needed to ensure they perform in the expected way. In this article we present a framework for such a monitoring system. The proposed system is designed and deployed at Mastercard. This system monitors other machine learning systems that are deployed for use in production. The monitoring system performs concept drift detection by tracking the machine learning system’s inputs and outputs independently. Anomaly detection techniques are employed in the system to provide automatic alerts. We also present results that demonstrate the value of the framework. The monitoring system framework and the results are the main contributions in this article.