An in-depth analysis of CUSUM algorithm for the detection of mean and variability deviation in time series

Abstract

Assessing and enhancing data quality in sensors networks is an important challenge. In fact, data recorded and sent by the sensors are often dirty, they contain noisy, erroneous and missing values. This can be due to many reasons such as sensor malfunction, uncalibrated sensor and low battery of the sensor or caused by external factors such as climatic conditions or interference. We focus in this paper on change detection in the time series data issued from sensors. We particularly address slow and gradual changes as they illustrate sensor calibration drift. For this purpose, we provide in this paper an in-depth analysis and improvement of the well known Cumulative Sum (CUSUM) control chart algorithm, as it is well adapted to small shifts detection. First, we discuss the choice of the different parameters of CUSUM in order to optimize its results based on the trade-off between the false positives and the Average Run Length (ARLδ) needed by the algorithm to detect a process mean shift of δ. A study of the variability of the Run Length (RLδ) is provided using simulation. Then, we introduce two improvements to the CUSUM algorithm: we propose an efficient method to estimate the starting point and the end point of the mean shift. Moreover, we adapt CUSUM to detect not only process mean deviation but also process variability deviation. All these improvements are validated by simulation and against real data stream issued from water flow-meters.