Hybrid engine for polymorphic shellcode detection

Abstract

Driven by the permanent search for reliable anomaly-based intrusion detection mechanisms, we investigated different options of neural network (NN) based techniques. A further improvement could be achieved by combining the best suited NN-based data mining techniques with a mechanism we call "execution chain evaluation". This means that disassembled instruction chains are processed by the NN in order to detect malicious code. The proposed detection engine was trained and tested in various ways. Examples were taken from all publicly available polymorphic shellcode engines as well as from self-designed engines. A prototype implementation of our sensor has been realized and integrated as a plug-in into the SNORT™ [13] intrusion detection system. © Springer-Verlag Berlin Heidelberg 2005.