A multi-stage detection technique for DNS-tunneled botnets

Abstract

Botnet communications are obfuscated within legitimate network protocols to avoid detection and remediation. Domain Name Service (DNS) is a protocol of choice to hide communication with Command & Control (C&C) servers, where botmasters tunnel these communications within DNS request and response. Since botnet communications are characterized by different features, botmasters may evade detection methods by modifying some of these features. This paper proposes a multi-staged detection approach for Domain Generation Algorithm (DGA) using domain fluxing, Fast Flux Service Network (FFSN), and encrypted DNS tunneled-based botnets using BRO Network Security Monitor. This approach is able to detect DNS-tunneled botnet communications by analyzing different techniques used to find C&C servers, and also using signature matching technique to detect DNS-tunneled SSH handshake between bots and C&C servers.