Intelligent clustering scheme for log data streams

Abstract

Mining patterns from the log messages is valuable for real-time analysis and detecting faults, anomaly and security threats. A data-streaming algorithm with an efficient pattern finding approach is more practical way to classify these ubiquitous logs. Thus, in this paper the authors propose a novel online approach for finding patterns in log data sets where a locally sensitive signature is generated for similar log messages. The similarity of these log messages is identified by parsing log messages and then, logically analyzing the signature bit stream associated with them. In addition to that the approach is intelligent enough to reflect the changes when a totally new log appears in the system. The validation of the proposed method is done by comparing F-measure of clustering results for labeled datasets and the word order matched percentage of the log messages in a cluster for unlabeled case with that of SLCT. © 2014 Springer-Verlag Berlin Heidelberg.