Evaluating the robustness of adversarial defenses in malware detection systems

TL;DR

This work addresses robustness of ML-based Android malware detectors in binary feature spaces against evasion. It introduces sigma-binary, a gradient-based adversarial attack tailored for discrete binary features, and Prioritized Binary Rounding (PBR) to produce valid binary perturbations with minimal changes. Through extensive experiments on MalScan, sigma-binary consistently outperforms existing attacks, exposing critical vulnerabilities in state-of-the-art defenses, including adversarial detectors and adversarially trained models. The results reveal a three-way trade-off among robustness, accuracy, and computational cost, and provide a principled evaluation framework to guide the development of more resilient binary-domain malware detectors.

Abstract

Machine learning is a key tool for Android malware detection, effectively identifying malicious patterns in apps. However, ML-based detectors are vulnerable to evasion attacks, where small, crafted changes bypass detection. Despite progress in adversarial defenses, the lack of comprehensive evaluation frameworks in binary-constrained domains limits understanding of their robustness. We introduce two key contributions. First, Prioritized Binary Rounding, a technique to convert continuous perturbations into binary feature spaces while preserving high attack success and low perturbation size. Second, the sigma-binary attack, a novel adversarial method for binary domains, designed to achieve attack goals with minimal feature changes. Experiments on the Malscan dataset show that sigma-binary outperforms existing attacks and exposes key vulnerabilities in state-of-the-art defenses. Defenses equipped with adversary detectors, such as KDE, DLA, DNN+, and ICNN, exhibit significant brittleness, with attack success rates exceeding 90% using fewer than 10 feature modifications and reaching 100% with just 20. Adversarially trained defenses, including AT-rFGSM-k, AT-MaxMA, improves robustness under small budgets but remains vulnerable to unrestricted perturbations, with attack success rates of 99.45% and 96.62%, respectively. Although PAD-SMA demonstrates strong robustness against state-of-the-art gradient-based adversarial attacks by maintaining an attack success rate below 16.55%, the sigma-binary attack significantly outperforms these methods, achieving a 94.56% success rate under unrestricted perturbations. These findings highlight the critical need for precise method like sigma-binary to expose hidden vulnerabilities in existing defenses and support the development of more resilient malware detection systems.

Figures (6)

• Figure 1: Attack Success Rate (ASR) versus perturbation budget $k$ (Hamming distance $d_H$) across four defense models. Only the most competitive attacks are shown for clarity.
• Figure 2: Hyperparameter robustness of the $\sigma$-binary attack across smoothing parameter $\sigma$ and initial sparsity threshold $\gamma_0$. Each cell shows the Attack Success Rate (ASR) at different perturbation budgets and the corresponding median normalized Hamming distance. Top panel: AT-rFGSM$^k$; bottom panel: AT-MaxMA.
• Figure 3: Effect of the sparsity adjustment factor $t$ on the $\sigma$-binary attack. Left: ASR$_\infty$ versus $t$ (log scale). Right: median normalized Hamming distance $\tilde{d}_{H,\mathrm{median}}$ versus $t$.
• Figure 4: Robustness evaluation curves (ASR vs. perturbation budget $k$) against oblivious attacks.
• Figure 5: Robustness evaluation curves (ASR vs. perturbation budget $k$) against adaptive attacks.