Communication-efficient surrogate quantile regression for non-randomly distributed system

Abstract

Distributed system has been widely used to solve massive data analysis tasks. This article targets on quantile regression on distributed system with non-randomly distributed massive data, and proposes a new communication-efficient surrogate quantile regression. Specifically, based on a small size random pilot sample collected from different worker machines, we approximate the global quantile regression as a surrogate one on the master machine, which relates to the local datasets only through their gradient vectors, and can overcome the non-randomly distributed nature. Then the resulting estimator can be obtained on the master, and the communication cost is greatly reduced, since the pilot sample and local gradients can be transferred conveniently. In theory, without any restrictive assumption about randomness, the established asymptotical results show that the proposed method works beautifully just as the data were stored on one single machine. Synthetic data and real world data evaluations are also used to illustrate the proposed method.