Latent-Space Contrastive Reinforcement Learning for Stable and Efficient LLM Reasoning

TL;DR

This paper addresses the difficulty of reliable, long-horizon reasoning in large language models by reframing reinforcement learning from token-space exploration to latent-space planning. The authors propose DeepLatent Reasoning (DLR), which uses a lightweight Assistant to sample latent reasoning trajectories in a continuous space of dimension $d$, filters them with a dual $R_{tot}=R_{corr}+\lambda R_{fmt}$ reward, and decodes only high-value trajectories with a frozen Main Model, thereby eliminating catastrophic forgetting. A contrastive regularization term promotes diversity among latent trajectories, enabling directed exploration without degrading pre-trained knowledge. Empirically, DLR achieves state-of-the-art stability and efficiency on GSM8K and MATH under the same compute budget, reducing main-model forward passes by about a factor of five while improving accuracy and reducing hallucinations, demonstrating a viable path toward scalable, reliable RL for LLM reasoning.

Abstract

While Large Language Models (LLMs) demonstrate exceptional performance in surface-level text generation, their nature in handling complex multi-step reasoning tasks often remains one of ``statistical fitting'' rather than systematic logical deduction. Traditional Reinforcement Learning (RL) attempts to mitigate this by introducing a ``think-before-speak'' paradigm. However, applying RL directly in high-dimensional, discrete token spaces faces three inherent challenges: sample-inefficient rollouts, high gradient estimation variance, and the risk of catastrophic forgetting. To fundamentally address these structural bottlenecks, we propose \textbf{DeepLatent Reasoning (DLR)}, a latent-space bidirectional contrastive reinforcement learning framework. This framework shifts the trial-and-error cost from expensive token-level full sequence generation to the continuous latent manifold. Specifically, we introduce a lightweight assistant model to efficiently sample $K$ reasoning chain encodings within the latent space. These encodings are filtered via a dual reward mechanism based on correctness and formatting; only high-value latent trajectories are fed into a \textbf{frozen main model} for single-pass decoding. To maximize reasoning diversity while maintaining coherence, we design a contrastive learning objective to enable directed exploration within the latent space. Since the main model parameters remain frozen during optimization, this method mathematically eliminates catastrophic forgetting. Experiments demonstrate that under comparable GPU computational budgets, DLR achieves more stable training convergence, supports longer-horizon reasoning chains, and facilitates the sustainable accumulation of reasoning capabilities, providing a viable path toward reliable and scalable reinforcement learning for LLMs.

Figures (2)

• Figure 1: The latent-space reinforcement learning pipeline of DeepLatent Reasoning (DLR).
• Figure 2: The Training pipeline of DeepLatent Reasoning (DLR).