Impact of environment on branch transfer of software

Abstract

Current intrusion detection approaches based on control flow integrity (CFI) can detect the majority of control flow hijacking attacks, but few of them take into account the impact of environment on CFI, so there may exist false alarms. In this paper, we have investigated systematically the impact of environment on branch transfer from time, space and mechanisms of Linux operating system. Moreover, we have presented finite state automata (FSA) to describe difference patterns caused by these environmental factors, and have exploited FSA-Stack model to detect these impacts. Finally, for some common applications (gzip, grep, tesseract, bzip2 etc.), we have leveraged a dynamic binary instrumentation tool Pin to record direct and indirect branch transfers produced by them and the shared libraries they depend on. The experimental results demonstrate that impact of environment on branch transfer exists universally and normally among usual applications, and the difference patterns of impacts can be beneficial to understand and mitigate the false alarms of CFI.