Deobfuscation of virtualization-obfuscated code through symbolic execution and compilation optimization

Abstract

Virtualization-obfuscation replaces native code in a binary with semantically equivalent and self-defined bytecode, which, upon execution, is interpreted by a custom virtual machine. It makes the code very difficult to analyze and is thus widely used in malware. How to deobfuscate such virtualization obfuscated code has been an important and challenging problem. We approach the problem from an innovative perspective by transforming it into a compilation optimization problem, and propose a novel technique that combines trace analysis, symbolic execution and compilation optimization to defeat virtualization obfuscation. We implement a prototype system and evaluate it against popular virtualization obfuscators; the results demonstrate that our method is effective in deobfuscation of virtualization-obfuscated code.