HAGEN: Unveiling Obfuscated Memory Threats via Hierarchical Attention-Gated Explainable Networks

Abstract

Memory resident malware, particularly fileless and heavily obfuscated types, continues to pose a major problem for endpoint defense tools, as these threats often slip past traditional signature-based detection techniques. Deep learning has shown promise in identifying such malicious activity, but its use in real Security Operations Centers (SOCs) is still limited because the internal reasoning of these neural network models is difficult to interpret or verify. In response to this challenge, we present HAGEN, a hierarchical attention architecture designed to combine strong classification performance with explanations that security analysts can understand and trust. HAGEN processes memory artifacts through a series of attention layers that highlight important behavioral cues at different scales, while a gated mechanism controls how information flows through the network. This structure enables the system to expose the basis of its decisions rather than simply output a label. To further support transparency, the final classification step is guided by representative prototypes, allowing predictions to be related back to concrete examples learned during training. When evaluated on the CIC-MalMem-2022 dataset, HAGEN achieved 99.99% accuracy in distinguishing benign programs from major malware classes such as spyware, ransomware, and trojans, all with modest computational requirements suitable for live environments. Beyond accuracy, HAGEN produces clear visual and numeric explanations—such as attention maps and prototype distances—that help investigators understand which memory patterns contributed to each decision, making it a practical tool for both detection and forensic analysis.