Features for behavioral anomaly detection of connectionless network buffer overflow attacks

Abstract

Buffer overflow (BO) attacks are one of the most dangerous threats in the area of network security. Methods for detection of BO attacks basically use two approaches: signature matching against packets’ payload versus analysis of packets’ headers with the behavioral analysis of the connection’s flow. The second approach is intended for detection of BO attacks regardless of packets’ content which can be ciphered. In this paper, we propose a technique based on Network Behavioral Anomaly Detection (NBAD) aimed at connectionless network traffic. A similar approach has already been used in related works, but focused on connection-oriented traffic. All principles of connection-oriented NBAD cannot be applied in connectionless anomaly detection. There is designed a set of features describing the behavior of connectionless BO attacks and the tool implemented for their offline extraction from network traffic dumps. Next, we describe experiments performed in the virtual network environment utilizing SIP and TFTP network services exploitation and further data mining experiments employing supervised machine learning (ML) and Naive Bayes classifier. The exploitation of services is performed using network traffic modifications with intention to simulate real network conditions. The experimental results show the proposed approach is capable of distinguishing BO attacks from regular network traffic with high precision and class recall.