N-gram against the machine: On the feasibility of the N-gram network analysis for binary protocols

Abstract

In recent years we have witnessed several complex and high-impact attacks specifically targeting "binary" protocols (RPC, Samba and, more recently, RDP). These attacks could not be detected by current - signature-based - detection solutions, while - at least in theory - they could be detected by state-of-the-art anomaly-based systems. This raises once again the still unanswered question of how effective anomaly-based systems are in practice. To contribute to answering this question, in this paper we investigate the effectiveness of a widely studied category of network intrusion detection systems: anomaly-based algorithms using n-gram analysis for payload inspection. Specifically, we present a thorough analysis and evaluation of several detection algorithms using variants of n-gram analysis on real-life environments. Our tests show that the analyzed systems, in presence of data with high variability, cannot deliver high detection and low false positive rates at the same time. © 2012 Springer-Verlag.