Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning

TL;DR

Llm reasoning remains error-prone despite strong mathematical capabilities; this work introduces Generative Adversarial Reasoner (GAR), an on-policy adversarial RL framework that co-trains a Reasoner with a slice-level Discriminator. By partitioning reasoning into complete slices and providing concise, on-policy feedback through GRPO-based updates, GAR delivers dense credit signals that improve stepwise reasoning and final correctness with controlled compute. Empirically, GAR yields consistent improvements across seven mathematical benchmarks (notably AIME and LiveMathBench-Hard) and supports flexible reward shaping for tasks like distillation and proof-based reasoning. This approach enhances robustness and sample efficiency in complex reasoning tasks and can be extended to broader reasoning and verification scenarios.

Abstract

Large language models (LLMs) with explicit reasoning capabilities excel at mathematical reasoning yet still commit process errors, such as incorrect calculations, brittle logic, and superficially plausible but invalid steps. In this paper, we introduce Generative Adversarial Reasoner, an on-policy joint training framework designed to enhance reasoning by co-evolving an LLM reasoner and an LLM-based discriminator through adversarial reinforcement learning. A compute-efficient review schedule partitions each reasoning chain into logically complete slices of comparable length, and the discriminator evaluates each slice's soundness with concise, structured justifications. Learning couples complementary signals: the LLM reasoner is rewarded for logically consistent steps that yield correct answers, while the discriminator earns rewards for correctly detecting errors or distinguishing traces in the reasoning process. This produces dense, well-calibrated, on-policy step-level rewards that supplement sparse exact-match signals, improving credit assignment, increasing sample efficiency, and enhancing overall reasoning quality of LLMs. Across various mathematical benchmarks, the method delivers consistent gains over strong baselines with standard RL post-training. Specifically, on AIME24, we improve DeepSeek-R1-Distill-Qwen-7B from 54.0 to 61.3 (+7.3) and DeepSeek-R1-Distill-Llama-8B from 43.7 to 53.7 (+10.0). The modular discriminator also enables flexible reward shaping for objectives such as teacher distillation, preference alignment, and mathematical proof-based reasoning.

Figures (3)

• Figure 1: Pass@1 accuracy on seven mathematical reasoning benchmarks. Our Generative Adversarial Reasoner (GAR) consistently improves over strong baselines across both Deepseek-R1-Distill-Qwen-7B and Deepseek-R1-Distill-Llama-8B. GAR achieves gains of +22.9% on AIME24 and +19.5% on AIME25 for the Llama backbone, as well as +35.3% on LiveMathBench-Hard for Qwen. These results demonstrate the robustness and generality of GAR in enhancing reasoning performance across diverse mathematical tasks (Tab. \ref{['tab:performance']}).
• Figure 2: GAR architecture. GAR is a reinforcement learning framework that jointly trains an LLM reasoner and a slice-level discriminator to improve reasoning accuracy and explainability. Unlike standard RL for LLMs, which computes reward signals mainly based on final answers, GAR leverages a discriminator to provide dense, slice-level rewards that evaluate the reasoning process at each intermediate step. More importantly, the discriminator is continuously co-evolved with the LLM reasoner, ensuring the reward signal remains aligned with the model’s current behavior and enabling sustained improvement and refinement of reasoning.
• Figure 3: Improving model without entropy collapse. Compared to DeepSeek-R1-Distill-Qwen-7B, our method raises AIME24 accuracy (+7.3), but maintains a comparable overall mean-entropy distribution (5.20% vs. 5.27%) (a) and tightens the "wrong" distribution (b), indicating better calibration with fewer extreme-uncertainty failures. Removing zero-entropy tokens (c) flips the ordering (entropy is higher on correct cases), revealing a selective-entropy behavior (decisive on deterministic spans, exploratory on decision-critical tokens) that aligns with the AIME24 accuracy gains.