Development of a Model for Malware Detection and Classification at the Byte Level Based on Transformer

Abstract

Malware threats have been a critical concern in cybersecurity, particularly due to the increasing complexity and constantly evolving variants that are difficult to detect using conventional signature-based or static rule-based methods. This research focused on developing a Transformer-based model at the byte level to detect and classify malware effectively and adaptively, thereby streamlining the analytical process without requiring specialized feature tokenization. The primary objective was to design and evaluate a Transformer model that captures universal and adaptive malware patterns directly from raw byte representations, enabling cross-platform applicability. A quantitative experimental approach was employed using three public datasets: Malware Detection PE-Based Analysis, MC-dataset-binary, and Malware.zip. Data processing involved byte embedding, dilated 1D convolution, multi-head self-attention, and attention pooling. Model optimization was conducted using AdamW with a combined scheduler, Stochastic Weight Averaging (SWA), random byte masking augmentation, and Mixup Embedding. Experimental results showed that the byte-level Transformer model achieved high classification accuracy across the three datasets, namely 99%, 92%, and 94%, respectively. These results demonstrate that a byte-level Transformer can effectively capture universal malware patterns in binary data, offering a flexible and highly accurate approach to developing resilient defenses against modern cyber threats.