Evading Deep Learning-Based Malware Detectors via Obfuscation: A Deep Reinforcement Learning Approach

TL;DR

This work shows that combining open-source obfuscation tools with a deep reinforcement learning agent can generate fully obfuscated adversarial malware capable of evading state-of-the-art detectors. The OBFU-mal framework extends a malware-testing environment with an expanded action space and a DQN-based agent to maximize evasion under a black-box setting. Empirical results on 3,456 samples demonstrate substantial evasion gains against MalConv (65.15%) and LGBM/EMBER (79.20%), outperforming nine benchmark AMG tactics, with ablation revealing the strongest gains from the combination of obfuscation and RL. The study highlights both the offensive implications for detector robustness and the defensive potential of adversarial retraining, while outlining avenues for more sophisticated obfuscation and defense-oriented research.

Abstract

Adversarial Malware Generation (AMG), the generation of adversarial malware variants to strengthen Deep Learning (DL)-based malware detectors has emerged as a crucial tool in the development of proactive cyberdefense. However, the majority of extant works offer subtle perturbations or additions to executable files and do not explore full-file obfuscation. In this study, we show that an open-source encryption tool coupled with a Reinforcement Learning (RL) framework can successfully obfuscate malware to evade state-of-the-art malware detection engines and outperform techniques that use advanced modification methods. Our results show that the proposed method improves the evasion rate from 27%-49% compared to widely-used state-of-the-art reinforcement learning-based methods.

Figures (4)

• Figure 1: Interface of Darkarmour
• Figure 2: RL agent-environment interaction
• Figure 3: The OBFU-mal architecture
• Figure 4: Evasion rates against ClamAV by OBFU-mal