Efficient malware detection using model-checking

Abstract

Over the past decade, malware costs more than $10 billion every year and the cost is still increasing. Classical signature-based and emulation-based methods are becoming insufficient, since malware writers can easily obfuscate existing malware such that new variants cannot be detected by these methods. Thus, it is important to have more robust techniques for malware detection. In our previous work [24], we proposed to use model-checking to identify malware. We used pushdown systems (PDSs) to model the program (this allows to keep track of the program's stack behavior), and we defined the SCTPL logic to specify the malicious behaviors, where SCTPL can be seen as an extension of the branching-time temporal logic CTL with variables, quantifiers, and predicates over the stack. Malware detection was then reduced to SCTPL model-checking of PDSs. However, in our previous work [24], the way we used SCTPL to specify malicious behaviors was not very precise. Indeed, we used the names of the registers and memory locations instead of their values. We show in this work how to sidestep this limitation and use precise SCTPL formulas that consider the values of the registers and memory locations to specify malware. Moreover, to make the detection procedure more efficient, we propose an abstraction that reduces drastically the size of the program model, and show that this abstraction preserves all SCTPL\X formulas, where SCTPL\X is a fragment of SCTPL that is sufficient to precisely characterize malware specifications. We implemented our techniques in a tool and applied it to automatically detect several malwares. The experimental results are encouraging. © 2012 Springer-Verlag.