Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chain Of Thought Compression: A Theoritical Analysis

Juncai Li, Ru Li, Yuxiang Zhou, Boxiang Ma, Jeff Z. Pan

TL;DR

This work addresses the challenge of compressing Chain-of-Thought (CoT) reasoning in large language models by introducing a formal theory of learning difficulty for implicit CoT. It defines Order-$r$ Interaction to quantify high-order logical dependencies and proves that the learning signal decays as $\Theta(m^{-r})$ when intermediate steps are compressed into latent states, creating barriers on irreducible problems. To rigorously evaluate these ideas, it introduces NatBool-DAG, a benchmark enforcing irreducible logical reasoning, and ALiCoT, a latent-state alignment framework that preserves reasoning signals by aligning latent tokens with explicit reasoning semantics. Empirical results show that ALiCoT delivers substantial speedups (up to $54.4\times$) while matching explicit CoT performance on complex tasks, and the 16-bit parity experiments confirm dramatic improvements in learning efficiency over naive implicit CoT baselines. Collectively, the work provides both a theoretical foundation and practical methodology for scalable, efficient CoT compression with broad implications for reasoning-intensive NLP tasks.

Abstract

Chain-of-Thought (CoT) has unlocked advanced reasoning abilities of Large Language Models (LLMs) with intermediate steps, yet incurs prohibitive computational costs due to generation of extra tokens. Recent studies empirically show that compressing reasoning steps into latent states, or implicit CoT compression, offers a token-efficient alternative. However, the mechanism behind CoT compression remains unclear. In this paper, we provide the first theoretical analysis of the difficulty of learning to internalize intermediate reasoning steps. By introducing Order-r Interaction, we prove that the learning signal for high-order logical dependencies exponentially decays to solve irreducible problem, where skipping intermediate steps inevitably leads to high-order interaction barriers. To empirically validate this, we introduce NatBool-DAG, a challenging benchmark designed to enforce irreducible logical reasoning and eliminate semantic shortcuts. Guided by our theoretical findings, we propose ALiCoT (Aligned Implicit CoT), a novel framework that overcomes the signal decay by aligning latent token distributions with intermediate reasoning states. Experimental results demonstrate that ALiCoT successfully unlocks efficient reasoning: it achieves a 54.4x speedup while maintaining performance comparable to explicit CoT.

Chain Of Thought Compression: A Theoritical Analysis

TL;DR

This work addresses the challenge of compressing Chain-of-Thought (CoT) reasoning in large language models by introducing a formal theory of learning difficulty for implicit CoT. It defines Order- Interaction to quantify high-order logical dependencies and proves that the learning signal decays as when intermediate steps are compressed into latent states, creating barriers on irreducible problems. To rigorously evaluate these ideas, it introduces NatBool-DAG, a benchmark enforcing irreducible logical reasoning, and ALiCoT, a latent-state alignment framework that preserves reasoning signals by aligning latent tokens with explicit reasoning semantics. Empirical results show that ALiCoT delivers substantial speedups (up to ) while matching explicit CoT performance on complex tasks, and the 16-bit parity experiments confirm dramatic improvements in learning efficiency over naive implicit CoT baselines. Collectively, the work provides both a theoretical foundation and practical methodology for scalable, efficient CoT compression with broad implications for reasoning-intensive NLP tasks.

Abstract

Chain-of-Thought (CoT) has unlocked advanced reasoning abilities of Large Language Models (LLMs) with intermediate steps, yet incurs prohibitive computational costs due to generation of extra tokens. Recent studies empirically show that compressing reasoning steps into latent states, or implicit CoT compression, offers a token-efficient alternative. However, the mechanism behind CoT compression remains unclear. In this paper, we provide the first theoretical analysis of the difficulty of learning to internalize intermediate reasoning steps. By introducing Order-r Interaction, we prove that the learning signal for high-order logical dependencies exponentially decays to solve irreducible problem, where skipping intermediate steps inevitably leads to high-order interaction barriers. To empirically validate this, we introduce NatBool-DAG, a challenging benchmark designed to enforce irreducible logical reasoning and eliminate semantic shortcuts. Guided by our theoretical findings, we propose ALiCoT (Aligned Implicit CoT), a novel framework that overcomes the signal decay by aligning latent token distributions with intermediate reasoning states. Experimental results demonstrate that ALiCoT successfully unlocks efficient reasoning: it achieves a 54.4x speedup while maintaining performance comparable to explicit CoT.
Paper Structure (38 sections, 3 theorems, 52 equations, 5 figures, 4 tables)

This paper contains 38 sections, 3 theorems, 52 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries and Setup
  4. The Parity Problem
  5. Transformer Model and Optimization Objective
  6. Chain-of-thought Learning Objectives
  7. Implicit Chain-of-Thought
  8. Theoretical results
  9. Order-$r$ Interaction and Signal Decomposition
  10. Empirical Analysis: Semantic Shortcuts vs. Logical Irreducibility
  11. Quantifying Interaction Orders
  12. Results and Analysis
  13. NatBool-DAG: A Benchmark for Irreducible Reasoning
  14. Dataset Construction
  15. Methodology: Implicit Reasoning via Latent State Alignment
  16. ...and 23 more sections

Key Result

Theorem 1

Consider the Transformer model defined in Section sec:transformer_model with a smooth activation function $\phi$. Let $\gamma_r = \phi^{(r)}(0)/(r-1)!$ be the interaction coefficient for order $r$. Let $\mathbb{I}_{j}^{(r)}$ indicate whether input $\bm{x}_j$ is part of the Order-$r$ Interaction set

Figures (5)

  • Figure 1: CoT compression impacts reasoning difficulty based on problem reducibility. For Reducible Problems (top), compression is effective as the model faces low difficulty in reconstructing reasoning relationships. Conversely, for Irreducible Problems (bottom), omitting intermediate steps removes essential computational states, triggering a complexity explosion that leads to failure.
  • Figure 2: Schematic of the k-Parity Problem. $\bm{x}_{17}, \cdots, \bm{x}_{22}$ represent intermediate reasoning steps, which are replaced by latent tokens in the case of implicit Chain-of-Thought.
  • Figure 3: Interaction Landscape. We compare Effective Interaction Density ($\rho_k$, bars, left log-axis) and Interaction Quality ($\phi_k$, lines, right axis). Commonsense tasks (left) show low-quality high-order interactions, implying reducibility. Mathematical tasks (right) maintain high quality, indicating irreducible logical dependencies. Note that we sampled fewer candidate tokens for higher-order interactions to mitigate combinatorial explosion.
  • Figure 4: Convergence Efficiency on 16-bit Parity. We plot the training steps required to reach $100\%$ accuracy. Imp.Base-1 suffers exponential growth, while ALiCoT maintains a flat curve.
  • Figure 5: Impact of compression steps on model performance and computational cost. Solid lines represent accuracy (left axis), while dashed lines indicate the average token cost (right axis). As the number of compressed steps increases, ALiCoT (red) demonstrates superior robustness compared to Imp. Base-1 (blue). Notably, at the highest compression level (Step 20), both methods incur a minimal cost of $\sim$8 tokens, yet ALiCoT outperforms the baseline by $>12\%$ (83.94% vs. 71.40%).

Theorems & Definitions (5)

  • Definition 1: Order-$r$ Interaction
  • Theorem 1: Generalized Gradient Signal Decomposition
  • Lemma 2: Concentration of Irrelevant Terms
  • proof
  • Lemma 3: Concentration of Latent Interactions