Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Capabilities and Fundamental Limits of Latent Chain-of-Thought

Jiaxuan Zou, Yaozhong Xiong, Yong Liu

TL;DR

This paper addresses the paradox in latent versus explicit chain-of-thought reasoning in large language models: latent, continuous representations enable broad exploration but impair precise symbolic computation, while explicit, discrete reasoning yields exact execution but poor exploration. It introduces the Symbolic Index as a central regulator of decisional certainty, triggering a fundamental trade-off between exploration and execution. The Coconut curriculum is shown to be theoretically necessary and sufficient for training latent reasoning, via a duality with the Conditional Information Bottleneck and imitation-learning arguments, ensuring convergence to expert-like policies. Empirically, the work demonstrates that Latent CoT excels at exploratory tasks but struggles on arithmetic-style problems, and that curriculum-guided training stabilizes learning, aligning latent reasoning with symbolic accuracy. Overall, the framework shifts design from binary architectural choices to adaptive control of decisional certainty, guiding future latent-symbolic hybrids toward robust, task-aware reasoning.

Abstract

Latent Chain-of-Thought (Latent CoT) models promise efficient reasoning via continuous representations, yet exhibit puzzling performance inconsistencies: excelling at exploration (ProsQA: 97.0%) but failing at computation (GSM8K: 34.1%). We reveal that this trade-off is governed by decisional certainty. Our contributions are threefold: (1) We theoretically characterize the fundamental Exploration-Execution Trade-off, proving that high certainty enables precise execution but inhibits exploration, while low certainty facilitates search but causes error accumulation. (2) We introduce the Symbolic Index--quantifying decisional commitment--as the core mechanism governing this trade-off and establish its causal relationship with both execution stability and exploration capability. (3) We prove that curriculum learning is theoretically necessary, as direct training provably fails due to distributional mismatch. Our framework shifts the design paradigm from binary architectural choices toward adaptive systems that dynamically regulate decisional certainty based on task demands.

Capabilities and Fundamental Limits of Latent Chain-of-Thought

TL;DR

This paper addresses the paradox in latent versus explicit chain-of-thought reasoning in large language models: latent, continuous representations enable broad exploration but impair precise symbolic computation, while explicit, discrete reasoning yields exact execution but poor exploration. It introduces the Symbolic Index as a central regulator of decisional certainty, triggering a fundamental trade-off between exploration and execution. The Coconut curriculum is shown to be theoretically necessary and sufficient for training latent reasoning, via a duality with the Conditional Information Bottleneck and imitation-learning arguments, ensuring convergence to expert-like policies. Empirically, the work demonstrates that Latent CoT excels at exploratory tasks but struggles on arithmetic-style problems, and that curriculum-guided training stabilizes learning, aligning latent reasoning with symbolic accuracy. Overall, the framework shifts design from binary architectural choices to adaptive control of decisional certainty, guiding future latent-symbolic hybrids toward robust, task-aware reasoning.

Abstract

Latent Chain-of-Thought (Latent CoT) models promise efficient reasoning via continuous representations, yet exhibit puzzling performance inconsistencies: excelling at exploration (ProsQA: 97.0%) but failing at computation (GSM8K: 34.1%). We reveal that this trade-off is governed by decisional certainty. Our contributions are threefold: (1) We theoretically characterize the fundamental Exploration-Execution Trade-off, proving that high certainty enables precise execution but inhibits exploration, while low certainty facilitates search but causes error accumulation. (2) We introduce the Symbolic Index--quantifying decisional commitment--as the core mechanism governing this trade-off and establish its causal relationship with both execution stability and exploration capability. (3) We prove that curriculum learning is theoretically necessary, as direct training provably fails due to distributional mismatch. Our framework shifts the design paradigm from binary architectural choices toward adaptive systems that dynamically regulate decisional certainty based on task demands.
Paper Structure (45 sections, 14 theorems, 100 equations, 5 figures, 1 table)

This paper contains 45 sections, 14 theorems, 100 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Chain-of-Thought (CoT)
  5. Latent Chain-of-Thought (Latent CoT)
  6. Coconut Training hao2024traininglargelanguagemodels
  7. Theoretical Analysis
  8. Duality with the Information Bottleneck
  9. Interpretation of $\beta(k)$.
  10. Latent CoT excels in exploration
  11. Latent CoT's Fragility in Symbolic Computation
  12. Unifying the Trade-off via the Symbolic Index
  13. Why Curriculum Learning Works
  14. The Training Paradox of Latent CoT
  15. Curriculum Learning as Controlled $\mathcal{I}_{\text{S}}$ Transition
  16. ...and 30 more sections

Key Result

Theorem 4.1

Under the constraint of any finite model capacity, the optimization objective of the Coconut curriculum (Preliminaries coconut_training) at stage $k$ can be rigorously reformulated as a constrained optimization problem. Its Lagrangian dual is precisely the Conditional Information Bottleneck (CIB) pr where $\beta(k) > 0$ ideally satisfies $\beta(k) \sim \frac{k}{M-k}$.

Figures (5)

  • Figure 1: Symbolic Index on GSM8K. Latent CoT (shown) maintains a low Symbolic Index ($\mathcal{I}_{\text{S}} \in [0.2, 0.5]$), indicating a dispersed probability distribution. It lacks the probability concentration ($\mathcal{I}_{\text{S}} \approx 1.0$) observed in Explicit CoT.
  • Figure 2: Symbolic Index on ProsQA. Latent CoT exhibits a stable, low $\mathcal{I}_{\text{S}}$ distribution across reasoning steps. This validates Theorem \ref{['thm:latent_cot_exploration_guarantee']}, showing that the model distributes probability mass across multiple latent paths rather than converging to a single token.
  • Figure 3: Performance Degradation under Noise. Standard CoT (orange) shows a threshold effect: performance is constant until $\sigma$ exceeds the decision margin. Latent CoT (blue) shows monotonic decay starting from $\sigma \approx 0$. This aligns with the derivation $A(\sigma) = \Phi(\frac{\Delta_l}{\sqrt{C}\sigma})$ in Appendix \ref{['app:accuracy_drop_derivation']}.
  • Figure 4: Task-Dependent Sensitivity. Latent CoT retains higher performance on ProsQA (orange) compared to GSM8K (blue) under identical noise levels. This indicates that exploratory tasks have a larger tolerance for state deviation than precise computational tasks.
  • Figure 5: PCA visualization of latent state embeddings for reasoning trajectories. In both (a) and (b), the latent thoughts (L1-L6) demonstrate a clear convergence, supporting our assumption of a compact, attracting set for latent states. This validates that the convergence dynamic is a general property of the latent reasoning process, not specific to a single task.

Theorems & Definitions (31)

  • Theorem 4.1: Coconut-CIB Duality
  • Theorem 4.3: Exploration Deficiency of CoT
  • Remark 4.4: Sampling-Based Methods
  • Theorem 4.5: Exploration Capability Guarantee of Latent CoT
  • Definition 4.6: Sub-decisional Perturbation
  • Theorem 4.7: Symbolic Integrity of CoT
  • Theorem 4.8: Compounding Error in Latent Computation
  • Definition 4.9: Symbolic Index $\mathcal{I}_{\text{S}}$
  • Definition 4.10: Logit Decision Margin $\Delta_l$
  • Theorem 4.11: Symbolic Stability Theorem
  • ...and 21 more