Structural feature based anomaly detection for packed executable identification

Abstract

Malware is any software with malicious intentions. Commercial anti-malware software relies on signature databases. This approach has proven to be effective when the threats are already known. However, malware writers employ software encryption tools and code obfuscation techniques to hide the actual behaviour of their malicious programs. One of these techniques is executable packing, which consists of encrypting the real code of the executable so that it is decrypted in its execution. Commercial solutions to this problem try to identify the packer and then apply the corresponding unpacking routine for each packing algorithm. Nevertheless, this approach fails to detect new and custom packers. Therefore, generic unpacking methods have been proposed which execute the binary in a contained environment and gather its actual code. However, these approaches are very time-consuming and, therefore, a filter step is required that identifies whether an executable is packed or not. In this paper, we present the first packed executable detector based on anomaly detection. This approach represents not packed executables as feature vectors of structural information and heuristic values. Thereby, an executable is classified as packed or not packed by measuring its deviation to the representation of normality (not packed executables). We show that this method achieves high accuracy rates detecting packed executables while maintaining a low false positive rate. © 2011 Springer-Verlag.