From Static to AI-Driven Detection: A Comprehensive Review of Obfuscated Malware Techniques

Abstract

The frequency of cyber attacks targeting individuals, businesses, and organizations globally has escalated in recent years. The evolution of obfuscated malware, designed to evade detection, has been unprecedented, employing new and sophisticated mechanisms to breach systems, steal sensitive data, and disrupt operations. This work advances research on obfuscated malware detection by offering a comprehensive review of studies conducted over the past decade on multiple platforms. In addition, the diversity of obfuscation techniques and the effectiveness of detection methods, such as static, dynamic, hybrid, and AI, are presented in a comparative manner. Furthermore, memory forensics, an often underexplored area, is of paramount importance for real-time analysis and the detection of advanced obfuscated malware. Hybrid analysis, which amalgamates the strengths of various approaches, emerges as a robust solution against obfuscated malware detection. The role of AI in detecting advanced ransomware, spyware, and fileless malware by enabling real-time detection and adaptive defenses against these increasingly prevalent threats is presented. In addition, a novel framework is proposed, combining Generative AI and digital twins to simulate and predict malware behavior, offering enhanced detection capabilities. This study synthesizes the findings of 76 approaches for the detection of obfuscated malware, incorporates cutting-edge technologies, and identifies open research challenges, such as ensuring scalability, enhancing generalization across platforms, and reducing resource requirements for constrained environments to guide future advancements in obfuscated malware detection.