Robust DDoS detection based on adversarial trained graph neural networks with proximal optimization

Abstract

The increasing sophistication of Distributed Denial-of-Service (DDoS) attacks, often orchestrated through large-scale, coordinated botnets, demands detection models that can capture complex network relationships and remain robust to adversarial evasion. In this work, we propose a novel intrusion detection framework that integrates Graph Neural Networks (GNN) and adversarial learning to exploit traffic structure and enhance model resilience jointly. Network traffic is modeled as a dynamic graph, where nodes represent IP addresses and edges denote traffic flows, enabling the extraction of relational features through message-passing mechanisms. To improve robustness against perturbed or stealthy attack patterns, we incorporate adversarial training by optimizing a min-max objective that minimizes empirical risk under worst-case input perturbations. The learning objective combines graph-based message aggregation with adversarial risk minimization, ensuring both topological awareness and resilience. We further adopt a proximal gradient optimization strategy to stabilize training across batches of heterogeneous graph data. Evaluations on CIC-IDS2017 and BoT-IoT datasets demonstrate that our method outperforms conventional CNN, LSTM, and standalone GNN baselines in both detection accuracy and adversarial robustness, making it a viable approach for real-world, resilient DDoS defense systems. Our experimental results show that the proposed model achieves an F1-score of 94.7% and a robust accuracy of 87.9% under PGD attack, markedly surpassing non-robust and non-stabilized baselines.