Reasoning about Reasoning: BAPO Bounds on Chain-of-Thought Token Complexity in LLMs

TL;DR

This work analyzes how many reasoning tokens a large language model must generate as input size grows, using the bounded attention prefix oracle (BAPO) framework to model information flow within transformers. It proves that three classic BAPO-hard tasks—Majority, Match3$_n$, and Reachability—require linear CoT length, i.e., $c(n)=\Omega(n)$, under constant-bandwidth BAPO-CoTs, and provides matching upper bounds via explicit constructions. The authors introduce a self-consistent variant, cBAPO, to avoid pathological input-doubling loopholes and show that the same lower bounds hold while enabling tighter, constructive upper bounds (e.g., Majority $O(n\log n)$, Match3$_n$ $O(n)$, Reachability $O(n^2)$). Experiments with frontier reasoning models corroborate the theory, revealing approximately linear scaling of internal reasoning tokens and showcasing the practical impact on inference-time costs, while also highlighting opportunities for efficiency gains through decomposition or new architectures. Overall, the paper offers a principled framework to reason about optimal reasoning length and its consequences for real-world scaling and compression of chain-of-thought in LLMs.

Abstract

Inference-time scaling via chain-of-thought (CoT) reasoning is a major driver of state-of-the-art LLM performance, but it comes with substantial latency and compute costs. We address a fundamental theoretical question: how many reasoning tokens are required to solve a problem as input size grows? By extending the bounded attention prefix oracle (BAPO) model--an abstraction of LLMs that quantifies the information flow required to solve a task--we prove lower bounds on the CoT tokens required for three canonical BAPO-hard tasks: binary majority, triplet matching, and graph reachability. We show that each requires $Ω(n)$ reasoning tokens when the input size is $n$. We complement these results with matching or near-matching upper bounds via explicit constructions. Finally, our experiments with frontier reasoning models show approximately linear reasoning token scaling on these tasks and failures when constrained to smaller reasoning budgets, consistent with our theoretical lower bounds. Together, our results identify fundamental bottlenecks in inference-time compute through CoT and offer a principled tool for analyzing optimal reasoning length.

Figures (3)

• Figure 1: Overview of our modeling approach. An LLM autoregressively generates reasoning tokens $s_1, \cdots s_m$ to eventually produce an answer $y$. Each token generation step is modeled using a bounded attention prefix oracle (BAPO) with constant bandwidths; see \ref{['sec:multistep_bapo']} for a formal definition. To produce each $s_t$, information in a BAPO can cross an arbitrary prefix/suffix cut only through an $a$-bit summary and up to $b$ attended tokens from the prefix; this constrained update repeats for steps $t=1, \ldots m+1$.
• Figure 2: Reasoning token usage and accuracy with different reasoning levels for GPT-5.2. With no reasoning, accuracy quickly degrades to random chance. All reasoning levels achieve near-perfect accuracy, but with approximately linear scaling in the number of tokens used. Gemini 2.5 Pro exhibits much higher token usage and worse accuracy on Majority; see \ref{['fig:gemini-reasoning']} for the non-truncated figure.
• Figure 3: Performance of GPT-5.2 with different CoT prompting approaches (all with reasoning_effort = none to disable internal reasoning). With fixed word limits, performance degrades with input length. Both the plain and algorithmic CoT approaches allow performance to stay high, at the expense of approximately linear token cost. The poor performance of CoT for Majority is due to refusals to do step-by-step counting (see \ref{['app:example-cots']}).

Theorems & Definitions (36)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6
• Definition 7
• Theorem 4.1
• Theorem 4.2
• Theorem 4.3