A Two-sample Kolmogorov-Smirnov-like test for big data

Abstract

Exploratory data analysis (EDA) is an important component of modern data analysis and data mining. The Big Data setting has made many traditional and useful EDA tools impractical and ineffective. Among such useful tools is the two-sample Kolmogorov-Smirnov (TS-KS) goodness-of-fit (GoF) test for assessing whether or not two samples arose from the same population. A TS-KS like testing procedure is constructed using chunked and averaged (CA) estimation paradigm. The procedure is named the TS-CAKS GoF test. Distributed and streamed implementations of the TS-CAKS procedure are discussed. The consistency of the TS-CAKS test is proved. A numerical study is provided to demonstrate the effectiveness and computational efficiency of the procedure.