Countering persistent kernel rootkits through systematic hook discovery

Abstract

Kernel rootkits, as one of the most elusive types of malware, pose significant challenges for investigation and defense. Among the most notable are persistent kernel rootkits, a special type of kernel rootkits that implant persistent kernel hooks to tamper with the kernel execution to hide their presence. To defend against them, an effective approach is to first identify those kernel hooks and then protect them from being manipulated by these rootkits. In this paper, we focus on the first step by proposing a systematic approach to identify those kernel hooks. Our approach is based on two key observations: First, rootkits by design will attempt to hide its presence from all running rootkit-detection software including various system utility programs (e.g., ps and ls). Second, to manipulate OS kernel control-flows, persistent kernel rootkits by their nature will implant kernel hooks on the corresponding kernel-side execution paths invoked by the security programs. In other words, for any persistent kernel rootkit, either it is detectable by a security program or it has to tamper with one of the kernel hooks on the corresponding kernel-side execution path(s) of the security program. As a result, given an authentic security program, we only need to monitor and analyze its kernel-side execution paths to identify the related set of kernel hooks that could be potentially hijacked for evasion. We have built a proof-of-concept system called HookMap and evaluated it with a number of Linux utility programs such as ls, ps, and netstat in RedHat Fedora Core 5. Our system found that there exist 35 kernel hooks in the kernel-side execution path of ls that can be potentially hijacked for manipulation (e.g., for hiding files). Similarly, there are 85 kernel hooks for ps and 51 kernel hooks for netstat, which can be respectively hooked for hiding processes and network activities. A manual analysis of eight real-world rootkits shows that our identified kernel hooks cover all those used in them. © 2008 Springer-Verlag Berlin Heidelberg.