Enhanced metamorphic techniques-a case study against havex malware

Abstract

Most of the commercial antiviruses are signature based, that is, they use existing database signature to detect the malware. Malware authors use code obfuscation techniques in their variety of malware with the aim of bypassing detection by antiviruses. Metamorphic malware change their internal structure hence evading signature based detection. For effective defense against the malware, their construction needs to be explored. This paper includes the study of different obfuscation techniques and possibilities of their extension with focus on garbage code insertion, instruction substitution and subroutine reordering. The objective is to make detection difficult by implementing subject techniques which bypass detection. Havex malware is used as a proof of concept for our antivirus evasion strategy. We have used Hidden Markov Models (HMM), which is a statistical based machine learning detection method, to test the effectiveness of our code morphing. This has shown the strength of our implemented obfuscation techniques.