Understanding privacy-utility tradeoffs in differentially private online active learning

Abstract

We consider privacy-preserving learning in the context of online learning. In settings where data instances arrive sequentially in streaming fashion, incremental training algorithms such as stochastic gradient descent (SGD) can be used to learn and update prediction models. When labels are costly to acquire, active learning methods can be used to select samples to be labeled from a stream of unlabeled data. These labeled data samples are then used to update the machine learning models. Privacy-preserving online learning can be used to update predictors on data streams containing sensitive information. The differential privacy framework quantifies the privacy risk in such settings. This work proposes a differentially private online active learning algorithm using stochastic gradient descent (SGD) to retrain the classifiers. We propose two methods for selecting informative samples. We incorporated this into a general-purpose web application that allows a non-expert user to evaluate the privacy-aware classifier and visualize key privacy-utility tradeoffs. Our application supports linear support vector machines and logistic regression and enables an analyst to configure and visualize the effect of using differentially private online active learning versus a non-private counterpart. The application is useful for comparing the privacy/utility tradeoff of different algorithms, which can be useful to decision makers in choosing which algorithms and parameters to use. Additionally, we use the application to evaluate our SGD-based solution and to show that it generates predictions with a superior privacy-utility tradeoff than earlier methods.