Generic unpacking method based on detecting original entry point

Abstract

In this paper, we focus on the problem of the unpacking of packed executables in a generic way. That is, we do not assume specific knowledge about the algorithms used to produce the packed executable to do the unpacking (i.e. we do not extract/create a reverse algorithm). In general, when launched, a packed executable will first reconstruct the code of the original program, write it down someplace in memory and then transfer the execution to that original code by assigning the Extended Instruction Pointer (EIP) to the so-called Original Entry Point (OEP) of the program. Accordingly, if we had a way to accurately identify that transfer event in the execution flow and thus the OEP, we could more easily extract the original code for analysis (cf. by inspecting the remaining code after the OEP was reached).We then propose an effective generic unpacking method based on the combination of two novel OEP detection techniques, one relying on the incremental measurement of the entropy of the information stored in the memory space assigned to the unpacking process, and the other on the incremental searching and counting of potential Windows API calls in that same memory space. © Springer-Verlag 2013.