Advancements in Machine Learning for Adaptive Intrusion Detection: A Comprehensive Review

Abstract

The stability and adaptability of the intrusion detection systems (IDS) are of prime importance in the current scenario of continuous combat between cyber attackers and defenders in cyber-crime activity. Those dedicated rule-based systems have made considerable advancements in detecting established patterns from attacks but severely lag behind when it comes to detection of emerging and evolving threats. Thus, this review article attempts to fill this research gap by synthesizing the recent methodologies and input from different studies regarding adapting intrusion detection today using machine learning techniques. The idea is to take into consideration the effectiveness of various ML algorithms along with feature selection methods and use of ensemble learning techniques for improving the IDS capability. The proposed methodology includes important procedures such as data loading and preprocessing, feature selection, exploratory data analysis, training and evaluating the model, applying a Bayesian Gaussian Mixture Model, performing threshold comparisons, and analyzing outcomes. Observed results from training six models, including Random Forest, MLP, LSTM, Logistic Regression, KNN, and Decision Tree, are reported. Among the six models trained, the Random Forest model emerged the best with an accuracy of 95.10%, an F1 score of 95.10%, precision of 95.11%, and recall of 95.10% while accruing a loss of 1.77. The models were trained and evaluated on the UNSW-NB15 dataset, a well-known network intrusion detection dataset. For each model, performance metrics such as accuracy, F1 score, precision, recall, and loss are reported. In addition, average Receiver Operating Characteristic Area under Curve analysis, optimal threshold analysis, and performance metric comparisons at various thresholds are included. The findings highlight the effectiveness of ML methods for adaptive intrusion detection and the importance of threshold selection for the optimal detection performance.