A Dual-Agent Adversarial Framework for Robust Generalization in Deep Reinforcement Learning

TL;DR

The paper tackles generalization gaps in deep reinforcement learning caused by reliance on irrelevant high-dimensional features. It proposes a dual-agent adversarial framework where two homogeneous agents perturb each other’s encoders to force robust, semantics-focused representations, while remaining compatible with policy optimizers like PPO. Theoretical analysis yields lower bounds linking training robustness to generalization and shows a monotonic improvement property under policy updates, connecting optimization dynamics to reduced sensitivity to extraneous features. Empirically, the approach yields substantial gains on the Procgen benchmark, particularly on hard-level tasks, across PPO and DAAC baselines, with minimal human priors and broad compatibility. Overall, the framework offers a principled, scalable path to more generalizable deep RL policies relevant to real-world variability.

Abstract

Recently, empowered with the powerful capabilities of neural networks, reinforcement learning (RL) has successfully tackled numerous challenging tasks. However, while these models demonstrate enhanced decision-making abilities, they are increasingly prone to overfitting. For instance, a trained RL model often fails to generalize to even minor variations of the same task, such as a change in background color or other minor semantic differences. To address this issue, we propose a dual-agent adversarial policy learning framework, which allows agents to spontaneously learn the underlying semantics without introducing any human prior knowledge. Specifically, our framework involves a game process between two agents: each agent seeks to maximize the impact of perturbing on the opponent's policy by producing representation differences for the same state, while maintaining its own stability against such perturbations. This interaction encourages agents to learn generalizable policies, capable of handling irrelevant features from the high-dimensional observations. Extensive experimental results on the Procgen benchmark demonstrate that the adversarial process significantly improves the generalization performance of both agents, while also being applied to various RL algorithms, e.g., Proximal Policy Optimization (PPO). With the adversarial framework, the RL agent outperforms the baseline methods by a significant margin, especially in hard-level tasks, marking a significant step forward in the generalization capabilities of deep reinforcement learning.

Figures (5)

• Figure 1: Overview of the adversarial process. Our method involves a game process between two homogeneous agents, as shown in the figure. The training samples are simultaneously input into the encoders of both agents, resulting in differing representations for the same observation. By adjusting the parameters of the two encoders, both agents aim to ensure that their own policy networks are robust to such differences while maximizing the influence of these differences on the other agent's policy network as much as possible. This minimax game process will eventually allow robust policy learning, preventing agents from overfitting to irrelevant features in high-dimensional observations, thereby enhancing generalization performance.
• Figure 2: The impacts of biases and reward functions on generalization.
• Figure 3: Adversarial policy learning.$\psi_1$ and $\pi_1$ represent the encoder and policy network of agent 1, while $\psi_2$ and $\pi_2$ represent the encoder and policy network of agent 2. $s_1$ and $s_2$ represent the training data for agent 1 and agent 2, respectively.
• Figure 4: Test performance curves of each method on eight hard-level Procgen games. Each agent is trained on 500 training levels for 50M environment steps and evaluated on the full distribution of levels. The mean and standard deviation is shown across three seeds.
• Figure 5: Train performance curves of each method on eight hard-level Procgen games.

Theorems & Definitions (8)

• Theorem 3.2: Generalization performance lower bound
• Theorem 3.3
• Theorem 3.4
• Theorem 3.5: Training performance lower bound
• Theorem 3.6: Monotonic improvement of training performance
• proof
• Lemma C.1
• proof