A Distributed Parallel Network Intrusion Detection System Based on Ray Framework With GPU Acceleration

Abstract

In the era of the Internet of Things and big data, the training of machine learning models has become increasingly demanding due to the vast amounts of data involved. Reducing training time and improving classification accuracy are essential. This article proposes a high-performance attack detection model (AE-XGBoost) based on the distributed data parallel processing framework-Ray. First, a solution called Dynamic Resource Adjustment for Model Training enhances training speed by dynamically adjusting resources, preventing resource idleness or overload, and ensuring optimal resource utilization at each stage. Second, the Dual-Link Loss Autoencoder algorithm is employed for feature mining, improving anomaly detection and enabling clear visualization of normal and anomalous data. Finally, the data parallel XGBoost method is applied for attack classification. Experimental results on five public large-scale datasets demonstrate that the proposed model outperforms several well-established benchmark classification models in both performance and accuracy.