Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens

TL;DR

The paper tackles the limitations of explicit Chain-of-Thought prompting by enabling reasoning in a continuous latent space. It introduces Latent Thoughts Tuning (LT-Tuning), which fuses contextual hidden states with predictive embeddings through a Context-Prediction Fusion mechanism and employs a confidence-driven, three-stage curriculum to dynamically switch between latent and explicit thinking. Stage 1 trains explicit CoT capabilities, Stage 2 learns when to insert latent tokens using uncertainty, and Stage 3 blends context and prediction to form stable latent inputs. Across model scales from 1B to 8B, LT-Tuning consistently improves mathematical reasoning benchmarks, mitigates feature collapse that plagues latent methods, and exhibits robust scaling without external assistant models.

Abstract

While explicit Chain-of-Thought (CoT) equips Large Language Models (LLMs) with strong reasoning capabilities, it requires models to verbalize every intermediate step in text tokens, constraining the model thoughts to the discrete vocabulary space. Recently, reasoning in continuous latent space has emerged as a promising alternative, enabling more robust inference and flexible computation beyond discrete token constraints. However, current latent paradigms often suffer from feature collapse and instability, stemming from distribution mismatches when recurrently using hidden states as the input embeddings, or alignment issues when relying on assistant models. To address this, we propose Latent Thoughts Tuning (LT-Tuning), a framework that redefines how latent thoughts are constructed and deployed. Instead of relying solely on raw hidden states, our method introduces a Context-Prediction-Fusion mechanism that jointly leveraging contextual hidden states and predictive semantic guidance from the vocabulary embedding space. Combined with a progressive three-stage curriculum learning pipeline, LT-Tuning also enables dynamically switching between latent and explicit thinking modes. Experiments demonstrate that our method outperforms existing latent reasoning baselines, effectively mitigating feature collapse and achieving robust reasoning accuracy.

Figures (6)

• Figure 1: Comparison of reasoning paradigms. Explicit CoT verbalizes all steps as text tokens. Coconut uses a fixed number of latent tokens from hidden states. Soft-Thinking constructs latent tokens via probability-weighted interpolation with entropy-based stopping. Assistant-based methods rely on external models. Our LT-Tuning dynamically interleaves text and latent tokens through confidence-driven insertion and Context-Prediction Fusion.
• Figure 2: Overview of the three-stage LT-Tuning framework. Stage 1: standard explicit CoT fine-tuning to establish reasoning capabilities. Stage 2: learning to generate latent tokens with confidence-driven insertion, where hidden states serve as the initial latent representations. Stage 3: Context-Prediction Fusion, which combines contextual history information (hidden states) with predicted semantic guidance (fused embeddings) to construct high-quality latent tokens.
• Figure 3: Average number of <thinking> tokens generated versus question difficulty across models of varying sizes. Difficulty is measured by the error rate of Llama-3.1-8B-Instruct over 5 sampling trials. Models generally demonstrate a positive correlation between question difficulty and the number of generated latent tokens, indicating that our method learns to adaptively scale latent reasoning effort based on problem complexity.
• Figure 4: Visualization of step-wise model entropy and attention weights on latent tokens for Llama-3.1-8B. Shaded regions indicate ±1 standard error. Generation steps beyond 400 are truncated for clarity.
• Figure 5: PCA visualization of latent token embeddings across different reasoning steps for intrinsic methods on Llama-3.1-8B (we only show four key steps here for a better view). Each point represents a different sample from the test set. Coconut (green) exhibits severe feature collapse, where latent tokens from different samples converge to nearly identical points after just two reasoning steps. LT-Tuning w/o Stage 3 (blue) shows initial exploration in early positions but gradually collapses to similar representations in later steps. LT-Tuning (red) maintains semantic diversity even at six latent tokens, demonstrating its effectiveness in mitigating feature collapse while preserving exploration capacity in the latent space.