A discrete event system based approach for obfuscated malware detection

Abstract

With the growing use and popularity of Internet among people, security threats such as viruses, worms etc., are also rapidly increasing. In order to detect and prevent such threats, many antivirus softwares have been created. Signature matching approach used to detect malwares can be easily thwarted by using code obfuscation techniques. In this paper, we propose a discrete event systems-based approach to detect obfuscated malwares in a system, taking Bagle. A as our test virus. Commonly used obfuscation techniques have been applied to bagle. We built DES models for a process under attack and normal conditions with system calls as events. Based on the system calls evoked by any process, our detector will determine its maliciousness by comparing it with both the models.