A real-time approach for detecting malicious executables

Abstract

In this paper, we develop a real-time algorithm to detect malicious portable executable (PE) files. The proposed algorithm consists of feature extraction, vector quantization, and a classifier named Attribute-Biased Classifier (ABC). We have collected a large data set of malicious PE files from the Honeynet project in the EG-CERT and VirusSign to train and test the proposed system. We first apply a feature extraction algorithm to remove redundant features. Then the most effective features are mapped into two vector quantizers. Finally, the output of the two quantizers are given to the proposed ABC classifier to identify a PE file. The results show that our algorithm is able to detect malicious PE file with 99.3% detection rate, 97% accuracy, 0.998 AUC, and less than 1% false positive rate. In addition, our algorithm consumes a fraction of seconds to test a portable executable file.