Data Management for ML-Based Analytics and Beyond

Abstract

The increasing capabilities of machine learning (ML) has enabled the deployment of ML methods in a variety of applications, ranging from unstructured data analytics to autonomous vehicles. Due to the volumes of data over which ML is deployed, it is infeasible for humans to monitor deployments: the Tesla fleet of vehicles produces exabytes of data and millions of hours of video per day. As a result, ML deployments can fail in unexpected and catastrophic ways.In this work, we highlight three important but underlooked aspects of ML deployment pipelines: (1) managing high-quality training data, (2) monitoring ML errors at deployment time, and (3) connecting end use to deployment algorithms. We first demonstrate that training labels are often erroneous, contrary to standard practice, even when labeled by leading vendors. We then demonstrate that standard methods of deploying ML methods can lead to downstream errors. As a first step toward addressing these issues, we review and contextualize two abstractions for finding errors in training data and deployments. We further describe how to improve algorithms for analytics queries as a case study for optimizing ML pipelines end to end. Problem statement This paper considers the problem of end-to-end machine learning (ML) deployments, from the collection of training data all the way to answering queries using ML models. We focus on errors in ML models and the data used to train them, along with algorithms for end-to-end uses of these models. Methods We provide simple abstractions, model assertions and learned observation assertions (LOA) to find errors that are pervasive in ML model deployments and the data used to train these ML models. We further implemented our abstractions in open-source APIs. Our abstractions and APIs are easy for those who are not experts in ML to use and deploy. Results We show that model assertions and LOA can be deployed in as few as 10 lines of code per assertion. They can find errors with up to 100% precision across domains ranging from video analytics, tabular data analytics, and translation. Significance It is standard in the literature to assume that training data is "gold" (i.e., 100% accurate) and that bulk ML model metrics such as accuracy properly reflect model performance. We show that these are not the case, even in widely studied settings. Our simple abstractions point towards methods of checking end-to-end deployments. We hope that future work builds on our APIs.