Algorithmic complexity vulnerability analysis of a stateful firewall

Abstract

Algorithmic complexity vulnerabilities are an opportunity for an adversary to conduct a sophisticated kind of attack i.e. on network infrastructure services. Such attacks take advantage of worst case time or space complexity of algorithms implemented on devices in their software. In this paper we address potential risks introduced by such algorithmic behavior in computer networks in particular on a stateful firewall. First we introduce the idea and theoretical background for the attack. We then describe in full detail a successfully conducted attack which takes advantage of the worst case computational complexity of O(n2) of a hash table data structure used to store active sessions. The attack at hand is initiated from a network protected by an stateful firewall router feature to a remote server causing a DoS (Denial of Service) on an industry grade router. Our experimental results using a real life network topology show that by generating undetected low bandwidth but malicious network traffic causing collisions in the firewall’s hash table we cause the firewall to become unreachable or even announce a segmentation fault and reboot itself.