Comparing API Call Sequence Algorithms for Malware Detection

Abstract

Malware became more and more sophisticated and increasingly difficult to detect, thanks to the use of evasion techniques, including anti-emulation, encapsulation, obfuscation, packing, anti-virtualization, and anti-debugger. New malware variants are generated by removing, replacing, and adding useless API calls to the malicious code. To face this increasing number of malware, it is necessary to design new detection methods, which are in charge of quickly analyzing large dataset and its variants. In this work, the sequence of state transitions performed by the applications during their execution are modeled by Markov chains, and used for malware classification. The implemented Markov chain-based detector is compared with the sequence alignment algorithm, which is widely used in the literature. The considered dataset includes 7.3 K malware and 1.2 K benign Windows applications collected over public datasets. Experimental results show that the Markov chain detector detects malware with up to 95% F-measure and outperforms detector based on sequence alignment.