Robust Adversarial Reinforcement Learning

TL;DR

This work addresses the challenge of generalization in reinforcement learning under model mismatch and data scarcity by introducing Robust Adversarial Reinforcement Learning (RARL), where a learned adversary applies destabilizing disturbances during training. By framing policy learning as a zero-sum game and alternating updates between a protagonist and an adversary, the method optimizes for robustness and worst-case performance, leveraging TRPO and deep networks. Empirical results on multiple OpenAI Gym MuJoCo tasks show that RARL yields higher mean rewards, reduced variance, and superior generalization to changes in mass and friction, as well as resilience to adversarial disturbances. The approach provides a practical pathway to more reliable policies for real-world robots operating under uncertain dynamics and sim-to-real gaps.

Abstract

Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL). However, most current RL-based approaches fail to generalize since: (a) the gap between simulation and real world is so large that policy-learning approaches fail to transfer; (b) even if policy learning is done in real world, the data scarcity leads to failed generalization from training to test scenarios (e.g., due to different friction or object masses). Inspired from H-infinity control methods, we note that both modeling errors and differences in training and test scenarios can be viewed as extra forces/disturbances in the system. This paper proposes the idea of robust adversarial reinforcement learning (RARL), where we train an agent to operate in the presence of a destabilizing adversary that applies disturbance forces to the system. The jointly trained adversary is reinforced -- that is, it learns an optimal destabilization policy. We formulate the policy learning as a zero-sum, minimax objective function. Extensive experiments in multiple environments (InvertedPendulum, HalfCheetah, Swimmer, Hopper and Walker2d) conclusively demonstrate that our method (a) improves training stability; (b) is robust to differences in training/test conditions; and c) outperform the baseline even in the absence of the adversary.

Figures (9)

• Figure 1: We evaluate RARL on a variety of OpenAI gym problems. The adversary learns to apply destabilizing forces on specific points (denoted by red arrows) on the system, encouraging the protagonist to learn a robust control policy. These policies also transfer better to new test environments, with different environmental conditions and where the adversary may or may not be present.
• Figure 2: Cumulative reward curves for RARL trained policies versus the baseline (TRPO) when tested without any disturbance. For all the tasks, RARL achieves a better mean than the baseline. For tasks like Hopper, we also see a significant reduction of variance across runs.
• Figure 3: We show percentile plots without any disturbance to show the robustness of RARL compared to the baseline. Here the algorithms are run on multiple initializations and then sorted to show the n$^{th}$ percentile of cumulative final reward.
• Figure 4: Percentile plots with a learned adversarial disturbance show the robustness of RARL compared to the baseline in the presence of an adversary. Here the algorithms are run on multiple initializations followed by learning an adversarial disturbance that is applied at test time.
• Figure 5: The graphs show robustness of RARL policies to changing mass between training and testing. For the InvertedPendulum the mass of the pendulum is varied, while for the other tasks, the mass of the torso is varied.