Modular behavior profiles in systems with shared libraries

Abstract

Modern computing environments depend on extensive shared libraries. In this paper, we propose monitoring the calls between those libraries as a new source of data for host-based anomaly detection. That is, we characterize an application by its use of shared library functions and characterize each shared library function by its use of (lower-level) shared libraries. This approach to intrusion detection offers significant benefits, especially in systems such as Windows, much of which is implemented above the kernel as dynamically linked libraries (DLLs). It localizes anomalies to particular code modules, facilitating anomaly analysis and assessment and discouraging mimicry attacks. It reduces retraining after system updates and enables training concurrent with detection. The proposed approach can be used with various techniques for modeling call sequences, including N-grams, automata, and techniques that consider parameter values. To demonstrate its potential, we have studied how a DLL-level profiling IDS would detect two recent attacks on Windows systems.