Comparative Analysis of SVM, Random Forest, and XGBoost for Cybersecurity Backdoor Attack Detection

Abstract

Remote Access Trojans (RATs) delivered via malicious email attachments pose a significant backdoor threat to the cyber security of organizations, allowing for continuing unauthorized access, lateral movement, and extensive data theft within corporate settings. This paper provides a comparative analysis of three commonly utilized machine learning classifiers such as Support Vector Machine (SVM), Random Forest (RF), and XGBoost focusing specifically on the detection of malicious email attachments containing RATs, with a clear intent to safeguard organizational data, as these attachments remain the main initial vector for infection. In order to detect malware email, experiments are conducted in a real-world-inspired dataset using five static file features (file size, entropy, macro count, presence of JavaScript, and suspicious API calls), an 80:20 stratified train-test split, standard scaling, and ANOVA F-test selection of the top four discriminative features. Hyperparameter tuning via GridSearchCV (for SVM) and balanced class weighting were applied to address class imbalance. Results show that tree-based ensembles significantly outperform SVM: Random Forest achieved the highest performance with 97.6% accuracy, 95.6% precision, 98.5% recall, and 0.970 F1-score, closely followed by XGBoost (97.4% accuracy, 0.967 F1-score), whereas the tuned SVM lagged at 72.2% accuracy and 0.680 F1-score. Confusion matrices and ROC curves (AUC ≈ 1.0 for both RF and XGBoost) confirm the superior discriminative power of ensemble methods in this security-critical task.