Automatic detection and analysis of encrypted messages in malware

Abstract

Encryption is increasingly used in network communications, especially by malicious software (malware) to hide its malicious activities and protect itself from being detected or analyzed. Understanding malware’s encryption schemes helps researchers better analyze its network protocol, and then derive the internal structure of the malware. However, current techniques of encrypted protocol analysis have a lot of limitations. For example, they usually require the encryption part being separated from message processing which is hardly satisfied in today’s malware, and they cannot provide detailed information about the encryption parameter such as the algorithm used and its secret key. Therefore, these techniques cannot fulfill the needs of today’s malware analysis. In this paper, we propose a novel and enhanced approach to automatically detect and analyze encryption and encoding functions within network applications. Utilizing dynamic taint analysis and data pattern analysis, we are able to detect encryption, encoding and checksum routines within the normal processing of protocol messages without prior knowledge of the protocol, and provide detailed information about its encryption scheme, including the algorithms used, secret keys, ciphertext and plaintext. We can also detect private or custom encryption routines made by malware authors, which can be used as signature of the malware. We evaluate our method with several malware samples to demonstrate its effectiveness.