Pruning Cannot Hurt Robustness: Certified Trade-offs in Reinforcement Learning

TL;DR

This work introduces a principled framework for pruning RL policies under adversarial observation perturbations by modeling state-adversarial MDPs (SA-MDPs). It proves that elementwise pruning monotonically tightens certified robustness bounds for Gaussian and categorical policies with Lipschitz networks and derives a three-term regret decomposition that separates clean performance, pruning-induced loss, and robustness gains. Empirically, pruning reveals reproducible sweet spots at moderate sparsity across MuJoCo tasks, achieving substantial robustness gains with minimal or even improved clean performance, and demonstrating that pruning can function as a structural intervention rather than mere compression. The results suggest pruning as a robust, practical tool to enhance RL reliability in adversarial settings, with potential for broader application and integration with adversarial training strategies.

Abstract

Reinforcement learning (RL) policies deployed in real-world environments must remain reliable under adversarial perturbations. At the same time, modern deep RL agents are heavily over-parameterized, raising costs and fragility concerns. While pruning has been shown to improve robustness in supervised learning, its role in adversarial RL remains poorly understood. We develop the first theoretical framework for certified robustness under pruning in state-adversarial Markov decision processes (SA-MDPs). For Gaussian and categorical policies with Lipschitz networks, we prove that element-wise pruning can only tighten certified robustness bounds; pruning never makes the policy less robust. Building on this, we derive a novel three-term regret decomposition that disentangles clean-task performance, pruning-induced performance loss, and robustness gains, exposing a fundamental performance--robustness frontier. Empirically, we evaluate magnitude and micro-pruning schedules on continuous-control benchmarks with strong policy-aware adversaries. Across tasks, pruning consistently uncovers reproducible ``sweet spots'' at moderate sparsity levels, where robustness improves substantially without harming - and sometimes even enhancing - clean performance. These results position pruning not merely as a compression tool but as a structural intervention for robust RL.

Figures (13)

• Figure 1: The SA-MDP framework. A victim agent receives a perturbed observation from an adversary trained to reduce its performance. The true state $s_t$ is emitted by the environment, tampered by the adversary, and then passed to the victim.
• Figure 2: Clean vs. robust frontiers under pruning. (Top) hopper: normalized clean and robust returns as pruning increases, across strategies. (Bottom) halfcheetah: analogous trends with higher pruning tolerance. All curves are normalized to the unpruned SA-trained policy; shaded regions denote $\pm$ one standard error across seeds.
• Figure 3: hopper under attack. Robustness gains are strongest against MAD and RS adversaries, consistent with pruning’s global Lipschitz guarantee. Improvements are smaller and less consistent against targeted Value-guided attacks. Appendix \ref{['app:figures']} shows analogous plots for halfcheetah and walker2d.
• Figure 4: Pruning vs. adversarial training (hopper). Pruning yields robustness gains in both regimes. SA-regularization sometimes provides additional improvements (notably under RS and Random), but the effect is uneven across attacks.
• Figure 5: Micropruning ablation. Updating masks in small, periodic increments (10--20 steps) leads to smoother curves and more reliable sweet spots than pruning every step.

Theorems & Definitions (6)

• Theorem 3.1: SA-MDP robustness improves under pruning
• Theorem 3.2: Unified regret under SA attack
• Theorem 3.3: Performance--robustness trade-off under pruning
• Lemma B.1: Local robustness bound
• Lemma B.2: Expected robustness gap
• Lemma B.3: Clean/attacked value drop under pruning