Detecting return-to-libc buffer overflow attacks using network intrusion detection systems

Abstract

There has been a significant amount of research recently into methods of protecting systems from buffer overflow attacks by detecting stack injected shell code. The majority of the research focuses on developing algorithms or signatures for detecting polymorphic and metamorphic payloads. However much of this problem has already been solved through the mainstream use of host based protection mechanisms e.g. Data Execution Prevention (DEP) and Address Space Randomization (ASLR). Many hackers are now using the more inventive attack methods e.g., return-to-libc, which do not inject shell code onto the stack and thus evade DEP and common shell code detection mechanisms. The purpose of this work is to propose a series of generic signatures that could be used to detect network born return-to-libc attacks. To this end we outline how we performed a return-to-libc network based attack, which bypasses DEP and common IDS signatures, before suggesting an efficient signature for detection of similar return-to-libc attacks. © 2010 IEEE.