Improving Drift Detection by Monitoring Shapley Loss Values

Abstract

Along the deployment of Machine Learning models rises an inherent need for monitoring, where model performances should be tracked as well as potential drifts. In a live environment, with evolving data, the risk is for the model to become ill-adapted for the given situation. The failure to detect drift while leading to a performance deterioration could also cause side effects due to model over-trust. Informing the user of any anomaly upon detection is the key to enabling any action. We propose Shap-ADWIN, a novel approach improving the performance of state-of-the-art drift detectors such as ADWIN by leveraging the information brought by Shapley Loss Values. While common practice is to monitor the evolution of the loss of models at most for every predicted instance, the proposed solution monitors each individual instance and features the Shapley Loss value. Whenever the loss is attributed more toward a given feature the information becomes more contrasted, which enables a better detection. Indeed the signal-to-noise ratio is higher on that feature and allows the detector to leverage that information. The opposite case being equal Shapley values that are just the Loss under-scaled for every feature. Moreover, noise over the output would be equally distributed along with each Shapley Loss value of every feature providing lower information to noise ratio and allows a more reliable detection. We provide: a restricted proof, experiments and source code. Results were obtained using synthetically generated data presenting diverse types of drift, showing the performance of Shap-ADWIN over ADWIN.