Detecting anomalies in cluster system using hybrid deep learning model

Abstract

Anomaly detection is of great importance for data centers. It could help operations discover system failures and perform root cause analysis. In recent years, deep learning has achieved great results in many fields. Therefore, people begin to pay attention to applying deep learning for automatic anomaly detection. Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) are two classical structures in deep learning, which could effectively detect anomalies from system logs. However, the existing CNN-based and LSTM-based models have their shortcomings. In this paper, we propose a novel hybrid model for detecting anomalies from system logs, which mainly consist of CNN and LSTM. Considering logs as natural language sequences, the hybrid model embeds logs first and extracts features from embedding vectors with a convolution layer. Then LSTM is used to automatically learn temporal log patterns from normal execution. The proposed model compares the log patterns that occur in practice with the log patterns it has learned before and detects anomalies when practical log patterns deviate from the model trained from log data under normal execution. To evaluate the performance of our model, we conduct experiments over large log data. The experiment results show that the hybrid model combines the advantages of CNN and LSTM models, which is an unsupervised model, reduces the requirements of training data set, and achieves great result in anomaly detection.