Network traffic pattern analysis using improved information theoretic co-clustering based collective anomaly detection

Abstract

Collective anomaly is a pattern in the data when a group of similar data instances behave anomalously with respect to the entire dataset. Clustering is a useful unsupervised technique to identify the underlying pattern in the data as well as anomaly detection. However, existing clustering based techniques have high false alarm rates and consider individual data instance behaviour for anomaly detection. In this paper, we formulate the problem of detecting DoS (Denial of Service) attacks as collective anomaly detection and propose a mathematically logical criteria for selecting the important traffic attributes for detecting collective anomaly. Information theoretic co-clustering algorithm is advantageous over regular clustering for creating more fine-grained representation of the data, however lacks the ability to handle mixed attribute data. We extend the co-clustering algorithm by incorporating the ability to handle categorical attributes which augments the detection accuracy of DoS attacks in benchmark KDD cup 1999 network traffic dataset than the existing techniques.