Diversification of system calls in linux binaries

Abstract

This paper studies the idea of using large-scale diversification to protect operating systems and make malware ineffective. The idea is to first diversify the system call interface on a specific computer so that it becomes very challenging for a piece of malware to access resources, and to combine this with the recursive diversification of system library routines indirectly invoking system calls. Because of this unique diversification (i.e. a unique mapping of system call numbers), a large group of computers would have the same functionality but differently diversified software layers and user applications. A malicious program now becomes incompatible with its environment. The basic flaw of operating system monoculture - the vulnerability of all software to the same attacks - would be fixed this way. Specifically, we analyze the presence of system calls in the ELF binaries. We study the locations of system calls in the software layers of Linux and examine how many binaries in the whole system use system calls. Additionally, we discuss the different ways system calls are coded in ELF binaries and the challenges this causes for the diversification process. Also, we present a diversification tool and suggest several solutions to overcome the difficulties faced in system call diversification. The amount of problematic system calls is small, and our diversification tool manages to diversify the clear majority of system calls present in standard-like Linux configurations. For diversifying all the remaining system calls, we consider several possible approaches.