Data stream clustering for application-layer DDoS detection in encrypted traffic

Abstract

Application-layer distributed denial-of-service attacks have become a serious threat to modern high-speed computer networks and systems. Unlike network-layer attacks, application-layer attacks can be performed using legitimate requests from legitimately connected network machines that make these attacks undetectable by signature-based intrusion detection systems. Moreover, the attacks may utilize protocols that encrypt the data of network connections in the application layer, making it even harder to detect an attacker’s activity without decrypting users’ network traffic, and therefore violating their privacy. In this paper, we present a method that allows us to detect various application-layer denial-of-service attacks against a computer network in a timely fashion. We focus on detection of the attacks that utilize encrypted protocols by applying an anomaly-detection-based approach to statistics extracted from network packets. Since network traffic decryption can violate ethical norms and regulations on privacy, the detection method proposed analyzes network traffic without its decryption. The method involves construction of a model of normal user behavior by analyzing conversations between a web server and its clients. The construction algorithm is self-adaptive and allows one to update the model every time a new portion of network traffic data becomes available for analysis. Once the model has been built, it can be applied to detect various application-layer types of denial-of-service attacks, including slow attacks, computational attacks, and more advanced attacks imitating normal web user behavior. The proposed technique is evaluated with realistic end user network traffic generated in our virtual network environment. Evaluation results show that these attacks can be properly detected, while the number of false alarms remains very low.