Remote Access Trojans Detection Using Convolutional and Transformer-based Deep Learning Techniques

Abstract

In recent years, the cybersecurity landscape has been marked by an increased focus on malware attacks, specifically on Trojan-type malware attack that poses a significant threat to Windows and Linux operating systems. This threat underscores the necessity for proactive prevention measures. The present study employs intelligent algorithms to detect malware, with a specific focus on identifying Remote Access Trojans (RATs). RATs are chosen for examination due to their persistent evolution, their increased number of attacks in the last period and, also, their discreet operation within executable files. The dataset for the algorithms is created by computing texture images over a large number of executable files infected with RATs. Various machine learning models, including VGG-16, VGG-19, and ResNet50, are explored for their effectiveness in identifying malicious programs. Additionally, this research delves into transformer architectures, such as the Vision Transformer (ViT), particularly in image classification tasks. Emphasizing the importance of integrating advanced machine learning techniques into cybersecurity efforts, this study aims to strengthen defense mechanisms against evolving cyber threats. Ongoing research is directed towards refining model performance and validating findings across diverse malware datasets.