Learning Robust Reasoning through Guided Adversarial Self-Play

TL;DR

This work tackles the brittleness of reinforcement-learned reasoning when conditioning is fallible by introducing GASP, a Guided Adversarial Self-Play framework. GASP builds a two-role game where a polluter corrupts the conditioning and an agent learns to diagnose and repair, all guided by verifiable final-answer rewards and reinforced by an in-distribution repair imitation objective. The approach yields robust improvements in recoverability, diagnosability, and reliability across multiple open-weight LLMs, often with preserved or enhanced clean accuracy, and it demonstrates that an adaptive adversarial curriculum emerges from self-play. The method also reveals that in-distribution guidance accelerates recovery learning and curtails representational drift, suggesting practical pathways to deploying robust reasoning systems in imperfect real-world contexts. Overall, GASP advances robust reasoning under noisy or misleading prompts by explicitly training context-diagnosis and repair capabilities without external labels or teachers, leveraging only outcome-based feedback.

Abstract

Reinforcement learning from verifiable rewards (RLVR) produces strong reasoning models, yet they can fail catastrophically when the conditioning context is fallible (e.g., corrupted chain-of-thought, misleading partial solutions, or mild input perturbations), since standard RLVR optimizes final-answer correctness only under clean conditioning. We introduce GASP (Guided Adversarial Self-Play), a robustification method that explicitly trains detect-and-repair capabilities using only outcome verification. Without human labels or external teachers, GASP forms an adversarial self-play game within a single model: a polluter learns to induce failure via locally coherent corruptions, while an agent learns to diagnose and recover under the same corrupted conditioning. To address the scarcity of successful recoveries early in training, we propose in-distribution repair guidance, an imitation term on self-generated repairs that increases recovery probability while preserving previously acquired capabilities. Across four open-weight models (1.5B--8B), GASP transforms strong-but-brittle reasoners into robust ones that withstand misleading and perturbed context while often improving clean accuracy. Further analysis shows that adversarial corruptions induce an effective curriculum, and in-distribution guidance enables rapid recovery learning with minimal representational drift.

Figures (6)

• Figure 1: GASP trains robust reasoners that resist misleading context, detect and repair errors, and answer reliably.
• Figure 2: A brittle model follows corruptions and fails, while a robust model detects the inconsistency, and repairs to reach the correct answer; corresponding results show GASP markedly improves recoverability, diagnosability, reliability, and clean accuracy.
• Figure 3: For fixes with similar lengths, teacher (GPT-5) singh2025openai generated fixes have higher negative log-likelihood than self-generated fixes, indicating that the model assigns lower probability to teacher fixes than to its own fixes.
• Figure 4: Guidance accelerates off-trajectory recovery and preserves retention in a maze analogue. (a,b) Trajectories during recovery training from a misleading start (blue) to the goal (red); the star denotes the original clean start used to train the initial rail policy. GRPO-only explores broadly and drifts, while in-distribution guidance quickly returns to the rail and exploits prior knowledge. (c) Success rate from the misleading start; (d) clean-start success during recovery training (retention). Curves show mean $\pm$ std over 5 seeds, each evaluated with 10 rollouts per checkpoint.
• Figure 5: Representation drift correlates with QA drops. (a,c) show PCA shift of layerwise mean hidden states on a fixed TruthfulQA / CommonsenseQA probe set. Each marker is a transformer layer $i$, plotted as $(\Delta m_{i,1},\, m_{i,2})$ after 2D PCA, where $\Delta m_{i,1}$ is the layer’s mean PC1 change from the base model (base lies on $\Delta\mathrm{PC1}{=}0$). The boxed $d^{(\ast)}$ is the Euclidean distance between base vs. post-training centroids in PCA space (global drift). (b,d) show methods with larger drift (e.g., GASP-T) suffer larger accuracy drops, while low-drift GASP largely preserves accuracy.

Theorems & Definitions (13)

• Lemma 1.1: First-order gain decomposition
• proof
• Proposition 1.3: Dominant scaling with $\pi_\theta(w^{\mathrm{fix}}\mid s^{\mathrm{poll}})$
• proof
• Corollary 1.4: Preference for in-distribution repairs
• proof
• Lemma 1.5: Binary-reward GRPO advantages
• proof
• Lemma 1.6: Gradient estimator given $K=k$
• proof