Investigating the Reliability of Machine Learning Predictions: Proposed Alterations to the DARE Model

Abstract

Machine Learning (ML) applications have spread across disciplines from biotechnology to supply chain security. Increases in data acquisition and warehousing allow development of sophisticated, production-ready ML pipelines. One such instance is the real-time prediction mechanism in a forward deployed environment to identify system errors or malfunctions, for example, Critical Infrastructures (CI). Although mechanisms of control exist to validate model performance, comprehensive work on reliability measures around ML predictions are lacking. Complex ML algorithms result in black-box solutions, where business leaders require details on the ML decision criteria. The absence of clear decision criteria will erode and undermine the reliability of deployable ML models, and thus present themselves as major barriers of adoption. Having thorough understanding of the decision criteria enhances adoption within complex environments. In this work we propose modifications to previously established reliability measure using detailed statistical methods which can enhance ML trustworthiness in real-time environments. Specifically, we propose variations in the Mahalanobis Distance calculation, focusing on feature variance as input for Laplacian decay approximation. The proposed modification will be evaluated across different ML paradigms, encompassing both continuous and categorical outcome modeling, to assess model performance and compare prediction reliability for identifying model drift. Additionally, future guidance will be given on proposed reliability model variations, enhancing model capability to actively adopt and account for task difficulty in each situation.