The Role of Learning in Attacking Intrusion Detection Systems

TL;DR

This work shows that ML-based NIDS are vulnerable to evasion by lightweight agents trained offline through reinforcement learning. By combining reconnaissance data collection, surrogate modeling, and a POMDP-based RL framework, the authors deploy an autonomous ingress-perturbation agent on compromised devices to evade victim NIDS in real time, without per-flow optimization. Across four NetFlow datasets and four victim detectors, the approach achieves up to 48.9% attack success with around 5.72 ms latency and 0.52 MB memory, and remains effective even under out-of-distribution and black-box threat models. The findings underscore a significant threat to flow-based IDS and motivate defense strategies that blend flow-level perturbation resilience with deeper packet inspection and robust detection features.

Abstract

Recent work on network attacks have demonstrated that ML-based network intrusion detection systems (NIDS) can be evaded with adversarial perturbations. However, these attacks rely on complex optimizations that have large computational overheads, making them impractical in many real-world settings. In this paper, we introduce a lightweight adversarial agent that implements strategies (policies) trained via reinforcement learning (RL) that learn to evade ML-based NIDS without requiring online optimization. This attack proceeds by (1) offline training, where the agent learns to evade a surrogate ML model by perturbing malicious flows using network traffic data assumed to be collected via reconnaissance, then (2) deployment, where the trained agent is used in a compromised device controlled by an attacker to evade ML-based NIDS using learned attack strategies. We evaluate our approach across diverse NIDS and several white-, gray-, and black-box threat models. We demonstrate that attacks using these lightweight agents can be highly effective (reaching up to 48.9% attack success rate), extremely fast (requiring as little as 5.72ms to craft an attack), and require negligible resources (e.g., 0.52MB of memory). Through this work, we demonstrate that future botnets driven by lightweight learning-based agents can be highly effective and widely deployable in diverse environments of compromised devices.

Figures (9)

• Figure 1: Attack Overview: The adversary uses network traffic data collected via reconnaissance to train an adversarial agent to be used in bot deployment to evade NIDS.
• Figure 2: Threat Models: Comparison of threat model scenarios dependent on access to the victim NIDS model and reconnaissance data.
• Figure 3: The three-phase attack pipeline proceeds in four sequential steps: (1) Data Collection: NetFlow traffic data is collected from a victim network environment (e.g., Cloud, IoT, Enterprise) and preprocessed into a dataset. (2) Environment Creation: Adversary-owned surrogate models are trained using standard ML libraries to approximate the decision boundaries of the victim NIDS. (3) Offline Training: An agent interacts with a POMDP environment to learn an evasion policy $\pi_{\theta}$ by observing feedback $(o_t,r_t)$ from the surrogate model and a reward function. (4) Deployment: The trained agent is deployed to perturb malicious flow from attack sources and to evade the victim NIDS.
• Figure 4: Victim NIDS Model Accuracy: F1 score on test NetFlow datasets of each machine learning model.
• Figure 5: Pre and Post Offline Training Performance: The increase in attack success rate (%) observed before RL training for each victim model across (Left) surrogate ML model architectures and (Right) NetFlow RL training datasets.