Semi-supervised learning for packed executable detection

Abstract

The term malware is coined to name any software with malicious intentions. One of the methods malware writers use for hiding their creations is executable packing. Packing consists of encrypting or hiding the real code of the executable in such a way that it is decrypted or unhidden in its execution. Widespread solutions to this issue first try to identify the packer used and next apply the corresponding unpacking routine for each packing algorithm. As it happens with malware obfuscations, this approach fails to detect new and custom packers. Generic unpacking is a technique that has been proposed to solve this issue. These methods usually execute the binary in a contained environment or sandbox to retrieve the real code of the packed executable. Because these approaches incur in a high performance overhead, a filter step is required to determine whether an executable is packed or not. Supervised machine-learning approaches have been proposed to handle this filtering step. However, the usefulness of supervised learning is far to be complete because it requires a high amount of packed and not packed executables to be identified and labelled previously. In this paper, we propose a new method for packed executable detection that adopts a well-known semi-supervised learning approach to reduce the labelling requirements of completely supervised approaches. We performed an empirical validation demonstrating that the labelling efforts are lower than when supervised learning is used while the system maintains high accuracy rates. © 2011 IEEE.