A Wolf in Sheep's Clothing: Practical Black-box Adversarial Attacks for Evading Learning-based Windows Malware Detection in the Wild

TL;DR

MalGuise presents a practical black-box adversarial framework against learning-based Windows malware detection by applying semantics-preserving, fine-grained call-based redividing transformations to the control-flow graph, guided by Monte Carlo Tree Search to optimize transformation sequences. It reconstructs realistic adversarial PE files that preserve semantics, achieving high attack success rates (>95% in most cases) across multiple detectors and substantial evasion against real-world antivirus products. The work demonstrates strong semantic preservation (SPR > 91%) and exposes tangible security risks in current defenses, while also evaluating potential defenses and ethical considerations. Overall, MalGuise highlights the need for more robust CFG-based malware detection and motivates future work on format-agnostic and dynamic-analysis defenses.

Abstract

Given the remarkable achievements of existing learning-based malware detection in both academia and industry, this paper presents MalGuise, a practical black-box adversarial attack framework that evaluates the security risks of existing learning-based Windows malware detection systems under the black-box setting. MalGuise first employs a novel semantics-preserving transformation of call-based redividing to concurrently manipulate both nodes and edges of malware's control-flow graph, making it less noticeable. By employing a Monte-Carlo-tree-search-based optimization, MalGuise then searches for an optimized sequence of call-based redividing transformations to apply to the input Windows malware for evasions. Finally, it reconstructs the adversarial malware file based on the optimized transformation sequence while adhering to Windows executable format constraints, thereby maintaining the same semantics as the original. MalGuise is systematically evaluated against three state-of-the-art learning-based Windows malware detection systems under the black-box setting. Evaluation results demonstrate that MalGuise achieves a remarkably high attack success rate, mostly exceeding 95%, with over 91% of the generated adversarial malware files maintaining the same semantics. Furthermore, MalGuise achieves up to a 74.97% attack success rate against five anti-virus products, highlighting potential tangible security concerns to real-world users.

Figures (12)

• Figure 1: A overview of learning-based malware detection.
• Figure 2: The overview framework of $\mathsf{MalGuise}$.
• Figure 3: The call-basedredividing redivides one basic block in the "LockBit 3.0" ransomware (i.e., Fig. \ref{['fig:one_basic_block:before']}) into a composite of three consecutive basic blocks (i.e., Fig. \ref{['fig:one_basic_block:after']}).
• Figure 4: The conceptual layout of the reconstructed adversarial Windows malware file for the "LockBit 3.0" ransomware.
• Figure 5: Frequency of the number of modified basic blocks of all adversarial malware that evades the three target systems.