Radmin: Early detection of application-level resource exhaustion and starvation attacks

Abstract

Software systems are often engineered and tested for functionality under normal rather than worst-case conditions. This makes the systems vulnerable to denial of service attacks, where attackers engineer conditions that result in overconsumption of resources or starvation and stalling of execution. While the security community is well familiar with volumetric resource exhaustion attacks at the network and transport layers, application-specific attacks pose a challenging threat. In this paper, we present Radmin, a novel system for early detection of application- level resource exhaustion and starvation attacks. Radmin works directly on compiled binaries. It learns and executes multiple probabilistic finite automata from benign runs of target programs. Radmin confines the resource usage of target programs to the learned automata, and detects resource usage anomalies at their early stages. We demonstrate the effectiveness of Radmin by testing it over a variety of resource exhaustion and starvation weaknesses on commodity off-the-shelf software.