A novel anomaly detection using small training sets

Abstract

Anomaly detection is an essential component of the protection mechanism against novel attacks. Traditional methods need very large volume of purely training dataset, which is expensive to classify it manually. A new method for anomaly intrusion detection is proposed based on supervised clustering and Markov chain model, which is designed to train from a small set of normal data. After short system call sequences are clustered, Markov chain is used to learn the relationship among these clusters and classify the normal or abnormal. The observed behavior of the system is analyzed to infer the probability that the Markov chain of the norm profile supports the observed behavior. Markov information source entropy and condition entropy are used to select parameters. The experiments have showed that the method is effective to detect anomalistic behaviors, and enjoys better generalization ability when a small number of training dataset is used only. © Springer-Verlag Berlin Heidelberg 2005.