Behavioral anomaly detection approach based on log monitoring

Abstract

Log monitoring has been an effective measure to detect anomalies in large-scale software systems. Many researches for anomaly detection are based on the analysis of log semantics or frequency features in a single time interval. In this paper, we present a new detection method which predicts the system state by detecting anomalous behaviors extracted from log messages. Our detection method consists of 2 major steps: First, preprocess log messages by log normalization and an efficient hierarchical clustering operation. Second, generate behavior pattern sets from clustered messages and assign an anomaly score to new log sequences according to the relation between the log sequences and corresponding behavior patterns. Experiments on real world log data show that our method can predict system anomalies with a high accuracy.