Machine learning-based real-time anomaly detection using data pre-processing in the telemetry of server farms

Abstract

Fast and accurate anomaly detection is critical in telemetry systems because it helps operators take appropriate actions in response to abnormal behaviours. However, recent techniques are accurate but not fast enough to deal with real-time data. There is a need to reduce the anomaly detection time, which motivates us to propose two new algorithms called AnDePeD (Anomaly Detector on Periodic Data) and AnDePed Pro. The novelty of the proposed algorithms lies in exploiting the periodic nature of data in anomaly detection. Our proposed algorithms apply a variational mode decomposition technique to find and extract periodic components from the original data before using Long Short-Term Memory neural networks to detect anomalies in the remainder time series. Furthermore, our methods include advanced techniques to eliminate prediction errors and automatically tune operational parameters. Extensive numerical results show that the proposed algorithms achieve comparable performance in terms of Precision, Recall, F-score, and MCC metrics while outperforming most of the state-of-the-art anomaly detection approaches in terms of initialisation delay and detection delay, which is favourable for practical applications.