NGBPA next generation botnet protocol analysis

Abstract

The command & control (c&c) protocols of botnets are moving away from plaintext IRC communicationt towards encrypted and obfuscated protocols. In general, these protocols are proprietary. Therefore, standard network monitoring tools are not able to extract the commands from the collected traffic. However, if we want to monitor these new botnets, we need to know how their protocol decryption works. In this paper we present a novel approach in malware analysis for locating the encryption and decryption functions in botnet programs. This information can be used to extract these functions for c&c protocols. We illustrate the applicability of our approach by a sample from the Kraken botnet. Using our approach, we were able to identify the encryption routine within minutes. We then extracted the c&c protocol encryption and decryption. Both are presented in this paper. © IFIP International Federation for Information Processing 2009.