Real-Time Ransomware Detection Using Reinforcement Learning Agents

Abstract

Traditional signature-based anti-malware tools often fail to detect zero-day ransomware attacks due to their reliance on known patterns. This paper presents a real-time ransomware detection framework that models system behavior as a Reinforcement Learning (RL) environment. Behavioral features—including file entropy, CPU usage, and registry changes—are extracted from dynamic analysis logs generated by Cuckoo Sandbox. A (DQN) agent is trained to proactively block malicious actions by maximizing long-term rewards based on observed behavior. Experimental evaluation across multiple ransomware families such as WannaCry, Locky, Cerber, and Ryuk demonstrates that the proposed RL agent achieves a superior detection accuracy, precision, and F1-score compared to existing static and supervised learning methods. Furthermore, ablation tests and latency analysis confirm the model’s robustness and suitability for real-time deployment. This work introduces a behavior-driven, generalizable approach to ransomware defense that adapts to unseen threats through continual learning.