DRMD: Deep Reinforcement Learning for Malware Detection under Concept Drift

TL;DR

This work tackles malware detection under concept drift by formulating detection as a one-step MDP (MD-MDP) and training a deep reinforcement learning agent (DRMD) that unifies classification, uncertainty-based active learning, and rejection. The agent uses PPO on a contextual-bandit setup with tailored rewards to balance accuracy, labeling cost, and drift adaptation, and is evaluated on two Android datasets with time-aware splits. Results show DRMD achieves substantial autonomous resilience to drift, with average AUT improvements of $8.66$ for classification-only and $10.90$ for classification-rejection policies, surpassing strong baselines and DCBs, and performing well under integrated AL and rejection strategies. These findings demonstrate the practical viability of DRL-based malware detectors that actively manage labeling, abstention, and adaptation in dynamic threat landscapes.

Abstract

Malware detection in real-world settings must deal with evolving threats, limited labeling budgets, and uncertain predictions. Traditional classifiers, without additional mechanisms, struggle to maintain performance under concept drift in malware domains, as their supervised learning formulation cannot optimize when to defer decisions to manual labeling and adaptation. Modern malware detection pipelines combine classifiers with monthly active learning (AL) and rejection mechanisms to mitigate the impact of concept drift. In this work, we develop a novel formulation of malware detection as a one-step Markov Decision Process and train a deep reinforcement learning (DRL) agent, simultaneously optimizing sample classification performance and rejecting high-risk samples for manual labeling. We evaluated the joint detection and drift mitigation policy learned by the DRL-based Malware Detection (DRMD) agent through time-aware evaluations on Android malware datasets subject to realistic drift requiring multi-year performance stability. The policies learned under these conditions achieve a higher Area Under Time (AUT) performance compared to standard classification approaches used in the domain, showing improved resilience to concept drift. Specifically, the DRMD agent achieved an average AUT improvement of 8.66 and 10.90 for the classification-only and classification-rejection policies, respectively. Our results demonstrate for the first time that DRL can facilitate effective malware detection and improved resiliency to concept drift in the dynamic setting of Android malware detection.

Figures (1)

• Figure 1: Overview figure showing the one-step classification of a sample using DRMD, with training components in blue.