Using finite automata to mine execution data for intrusion detection: A preliminary report

Abstract

The use of program execution traces to detect intrusions has proven to be a successful strategy. Existing systems that employ this approach are anomaly detectors, meaning that they model a program’s normal behavior and signal deviations from that behavior. Unfortunately, many program-based exploits of NT systems use specialized malicious executables. Anomaly detection systems cannot deal with such programs because there is no standard of “normalcy” that they deviate from. This paper is a preliminary report on an attempt to remedy that situation. We report on a prototype system that learns to identify specific program behaviors. Though the goal is to identify malicious behavior, in this paper we report on experiments seeking to identify the behavior of the web-browser, since we did not have enough exemplars of malicious behavior to use as training data. Using automatically generated finite automata, we search for features in execution traces that allow us to distinguish browsers from other programs. In our experiments, we find that this technique does, in fact, allow us to distinguish traces Internet Explorer from traces of programs that are not web browsers, after training with Netscape and a different set of non-browsers.