Image-Based Malware Detection Using α-Cuts and Binary Visualisation

Abstract

Image conversion of malicious binaries, or binary visualisation, is a relevant approach in the security community. Recently, it has exceeded the role of a single-file malware analysis tool and has become a part of Intrusion Detection Systems (IDSs) thanks to the adoption of Convolutional Neural Networks (CNNs). However, there has been little effort toward image segmentation for the converted images. In this study, we propose a novel method that serves a dual purpose: (a) it enhances colour and pattern segmentation, and (b) it achieves a sparse representation of the images. According to this, we considered the R, G, and B colour values of each pixel as respective fuzzy sets. We then performed (Formula presented.) -cuts as a defuzzification method across all pixels of the image, which converted them to sparse matrices of 0s and 1s. Our method was tested on a variety of dataset sizes and evaluated according to the detection rates of hyperparameterised ResNet50 models. Our findings demonstrated that for larger datasets, sparse representations of intelligently coloured binary images can exceed the model performance of unprocessed ones, with 93.60% accuracy, 94.48% precision, 92.60% recall, and 93.53% f-score. This is the first time that (Formula presented.) -cuts were used in image processing and according to our results, we believe that they provide an important contribution to image processing for challenging datasets. Overall, it shows that it can become an integrated component of image-based IDS operations and other demanding real-time practices.