ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluation

TL;DR

This work tackles the reward bottleneck in RL-based reasoning for large language models by proposing ALIVE, a unified self-supervised framework that jointly constructs tasks, solves them, and critiques the results within a single policy. It replaces external rewards with self-generated, instructive verbal feedback to internalize reasoning criteria, enabling scalable intrinsic alignment across domains. Empirical evaluations across mathematics, coding, and general reasoning demonstrate improved accuracy, cross-domain generalization, and higher self-correction rates under identical data and compute, with performance approaching or matching oracle-guided conditions. Collectively, ALIVE suggests a scalable path toward intrinsic reasoning capability and domain-general alignment without human supervision.

Abstract

The quest for expert-level reasoning in Large Language Models (LLMs) has been hampered by a persistent \textit{reward bottleneck}: traditional reinforcement learning (RL) relies on scalar rewards that are \textbf{costly} to scale, \textbf{brittle} across domains, and \textbf{blind} to the underlying logic of a solution. This reliance on external, impoverished signals prevents models from developing a deep, self-contained understanding of reasoning principles. We introduce \textbf{ALIVE} (\emph{Adversarial Learning with Instructive Verbal Evaluation}), a hands-free alignment framework that moves beyond scalar reward optimization toward intrinsic reasoning acquisition. Grounded in the principle of \emph{Cognitive Synergy}, ALIVE unifies problem posing, solving, and judging within a single policy model to internalize the logic of correctness. By coupling adversarial learning with instructive verbal feedback, ALIVE enables models to internalize evaluative criteria directly from raw corpora, effectively transforming external critiques into an endogenous reasoning faculty. Empirical evaluations across mathematical reasoning, code generation, and general logical inference benchmarks demonstrate that ALIVE consistently mitigates reward signal limitations. With identical data and compute, it achieves accuracy gains, markedly improved cross-domain generalization, and higher self-correction rates. These results indicate that the reasoning trinity fosters a self-sustaining trajectory of capability growth, positioning ALIVE as a scalable foundation for general-purpose reasoning alignment without human-in-the-loop supervision.

Figures (2)

• Figure 1: Overview of the ALIVE framework. A unified policy model $\pi_\theta$ alternates between three cognitive roles in a self-contained learning cycle: (1) Constructor masks critical spans in raw text to create tasks, (2) Solver generates reasoning trajectories for these tasks, and (3) Reviewer critiques its own solutions and provides both verbal feedback and soft rewards. The model parameters are updated by combining signals from all three roles, forming a closed-loop self-improvement system.
• Figure 2: Training Dynamics. A comparison of the Constructor's reward, Policy Entropy, and FCP Loss between the fully autonomous ALIVE-Self (Red) and the Oracle-guided ALIVE-Oracle (Black).