Metamorphic malware detection using LLVM IR and hidden Markov model

Abstract

This paper proposes a new method to detect metamorphic malware with the help of hidden Markov model and LLVM intermediate representation. The new approach improves the accuracy of HMM by simplifying various uncertain transformations present in the metamorphic code with the help of conversion of these instructions into the LLVM IR. Due to conversion of the unstructured assembly language code into the simplified LLVM IR, many of the code obfuscations are reversed and thus simplified form of instructions are generated. We can easily detect the remaining transformations or other unknown probabilistic states which HMM undergoes. Conversion to LLVM IR increases the predictability of HMM and also the probability to successfully detect other hidden states of malwares. Hence, this approach to first convert code into IR and then test the IR on HMM increases the probability of successful detection of metamorphic malwares.