Micro-signatures: The effectiveness of known bad N-grams for network anomaly detection

Abstract

Network intrusion detection is broadly divided into signature and anomaly detection. The former identifies patterns associated with known attacks and the latter attempts to learn a ‘normal’ pattern of activity and alerts when behaviors outside of those norms is detected. The n-gram methodology has arguably been the most successful technique for network anomaly detection. In this work we discover that when training data is sanitized, n-gram anomaly detection is not primarily anomaly detection, as it receives the majority of its performance from an implicit non-anomaly subsystem, that neither uses typical signatures nor is anomaly based (though it is closely related to both). We find that for our data, these “micro-signatures” provide the vast majority of the detection capability. This finding changes how we understand and approach n-gram based ‘anomaly’ detection. By understanding the foundational principles upon which it operates, we can then better explore how to optimally improve it.