A Hybrid Anomaly Detection Approach for Obfuscated Malware

Abstract

With the rapid evolution of malicious software, cyber threats have become increasingly sophisticated, em-ploying advanced obfuscation techniques to evade traditional detection methods. This study presents a hybrid anomaly detection approach applied to obfuscated malware. Even though there is a large body of research in this field, existing malware detection techniques have drawbacks, such as requiring large amounts of data, trustworthiness (imprecise results) of algorithms, and advanced obfuscation. There is a need to employ solid and efficient techniques for mal-ware detection to overcome these challenges. This paper proposes a hybrid approach, combining an autoencoder with traditional machine-learning methods to create an efficient malware detection framework. We used the malware memory dataset (MalMemAnalysis-2022) to evaluate this framework. The experimental results show our proposed approach can detect obfuscated malware when a deep autoencoder used for feature learning is combined with logistic regression. It is extremely fast with an Accuracy, Detection Rate (DR), Matthew Correlation Coefficient(MCC), and Statistical Parity Difference (SPD) of 99.97%, 99.98%, 99.93%, and 0.03%, respectively.