Detection and Analysis of Malicious Software Using Machine Learning Models

Abstract

The continuous evolution of malware poses a significant challenge in cybersecurity, adapting to technological advancements despite implemented security measures. This paper introduces an innovative approach to enhance the detection of obfuscated malware through the integration of machine learning (ML). Utilizing a real-world dataset of prevalent malware types such as spyware, ransomware, and trojan horses, our study addresses the evolving challenges of cybersecurity. In this study, we evaluate the performance of ML algorithms for obfuscated malware detection using the CIC-MalMem-2022 dataset. Our analysis encompasses binary and multi-class classification tasks under various experimental conditions, including percentage splits and 10-fold cross-validation. The evaluated algorithms include Random Tree (RT), Random Forest (RF), J-48 (C4.5), Naive Bayes (NB), and XGBoost. Experimental results demonstrate the effectiveness of RF, J-48, and XGBoost in achieving high accuracy rates across different classification tasks. NB also shows competitive performance but faces challenges in handling imbalanced datasets and multi-class classification. Our findings highlight the importance of employing advanced ML techniques for enhancing obfuscated malware detection capabilities and provide valuable insights for cybersecurity practitioners and researchers. Future research directions include fine-tuning model hyperparameters, exploring ensemble learning approaches, and expanding evaluation to diverse datasets and real-world scenarios.