A scalable MLOps-integrated intrusion detection framework using Bi-GRU and attention mechanisms with Kubeflow and KServe

Abstract

Highly motivated, sophisticated cyberattacks that target cloud capabilities with Internet of Things integrations require advanced, real-time, intelligent, scalable, and efficient Intrusion Detection Systems. This research proposes a novel Intrusion Detection Network (IDNet) architecture that utilises deep learning frameworks, combining Bidirectional Gated Recurrent Units (Bi-GRUs) with Attention Mechanisms to construct complex temporal dependencies in traffic while emphasising critical traffic instructions, pattern recognition, and other vital tasks. IDNet was implemented and tested on the Coburg Intrusion Detection Data Sets (CIDDS), which demonstrated superior performance compared to GRU, Attention-GRU, and Bi-GRU baselines in terms of accuracy and robustness. The proposed pipeline is implemented using Kubeflow Pipelines for Model training automation, Katib for hyperparameter optimisation, and MLflow/Kubeflow Metadata for model version control. Real-time inference is served using IDNet’s deployment on KServe, and performance is optimised with TorchServe and TensorRT. Through Grafana and Prometheus, observability is continuous and dynamic for metrics such as latency, throughput, false positive rate, marker shedding, and others. Adaptation to new changes is facilitated by the Population Stability Index, which initiates automatic retraining, ensuring defence against emerging threats. The solution is built on and works with Amazon Elastic Kubernetes Service. Integrated with Kafka/NATS, real-time traffic ingestion uses them for injection. After retraining the FPR IDS, IDNet achieved 98.90% accuracy alongside a 43.68% drop in latency and a 37.19% reduction in FPR. The data shows experimental results. Using comparative evaluation with the most advanced existing models validates IDNet’s efficiency as a real-time Intrusion Detection System for complex and high-traffic network environments.