Evaluating the robustness of adversarial defenses in malware detection systems

Abstract

Machine learning-based malware detectors effectively identify malicious Android applications, but remain highly vulnerable to evasion attacks, where minor binary feature perturbations can evade detection. Despite ongoing progress, the lack of comprehensive evaluation frameworks limits a clear understanding of defense robustness. This study introduces two contributions for adversarial evaluation in binary-constrained malware detection. First, we propose Prioritized Binary Rounding, an efficient technique that converts continuous perturbations into binary features while preserving a high attack success rate and minimal modification budgets. Second, we present the σ-binary attack, a gradient-based method tailored for binary domains, capable of producing sparse yet highly effective perturbations. Experimental results in the MalScan dataset show that σ-binary achieves consistently superior and stable performance. It surpasses state-of-the-art binary-domain approaches against both robust and non-robust defenses. Advanced defenses such as KDE, DLA, DNN+, and ICNN, when evaluated under the proposed σ-binary attack, exhibit over 90% attack success with fewer than 10 modified features and reach 100% under a 20-feature budget. Even PAD-SMA, previously reported to retain 83% accuracy, is compromised by σ-binary with 36.3% success under 10-features and 94.6% with no feature-budget constraints. These results highlight the need for precise evaluation tools such as the proposed σ-binary attack to uncover latent weaknesses and guide the design of truly robust malware detectors.