Lost in Transmission: When and Why LLMs Fail to Reason Globally

TL;DR

The paper investigates why LLMs fail at global reasoning by proposing the bounded attention prefix oracle (BAPO), a bandwidth-limited abstraction of transformer communication. It establishes theoretical BAPO-hardness for several natural problems, demonstrates how chain-of-thought (CoT) can reduce bandwidth requirements and even yield Turing-completeness under CoT, and provides empirical evidence showing LLMs struggle on BAPO-hard tasks while benefiting from CoT under sufficient reasoning budgets. The findings offer a principled lens to diagnose LLM failures, with practical implications for mitigation strategies such as tool use, inference-time scaling, and reasoning-focused training, as well as guiding future architectural developments to alleviate bandwidth bottlenecks.

Abstract

Despite their many successes, transformer-based large language models (LLMs) continue to struggle with tasks that require complex reasoning over large parts of their input. We argue that these failures arise due to capacity limits on the accurate flow of information within LLMs. To formalize this issue, we introduce the bounded attention prefix oracle (BAPO) model, a new computational framework that models bandwidth constraints on attention heads, the mechanism for internal communication in LLMs. We show that several important reasoning problems like graph reachability require high communication bandwidth for BAPOs to solve; we call these problems BAPO-hard. Our experiments corroborate our theoretical predictions: GPT-4o, Claude, and Gemini succeed on BAPO-easy tasks and fail even on relatively small BAPO-hard tasks. BAPOs also reveal another benefit of chain of thought (CoT): we prove that breaking down a task using CoT can turn any BAPO-hard problem into a BAPO-easy one. Our results offer principled explanations for key LLM failures and suggest directions for architectures and inference methods that mitigate bandwidth limits.

Figures (13)

• Figure 1: We conjecture that LLMs have a limit on their effective bandwidth, which we illustrate above by constrained information flow across one particular prefix--suffix split of the input. The BAPO model quantifies the communication bandwidth needed for transformers with causal attention to solve a problem. Index requires low communication bandwidth and LLMs solve it without issue; Reachability requires high bandwidth and LLMs struggle with it.
• Figure 2: A simplified view of a transformer and our bounded attention prefix oracle (BAPO) model.
• Figure 3: BAPO-hard problems (bottom row) show much larger drops in accuracy compared to BAPO-easy problems (top row). Not even large LLMs can solve BAPO-hard problems at length 200 with an accuracy above random guessing.
• Figure 4: Adding CoT can help LLMs do better on BAPO-hard problems, but substantial performance drops still occur with limited CoT budget (soft limit of 250 words for non-reasoning models). Without imposing a limit on their internal reasoning, o3 and, to a lesser extent, Gemini 2.5 Flash perform extremely well (see \ref{['app:cot-experiments']} for their CoT token counts, often in the thousands).
• Figure 5: There is good evidence that BAPO-difficulty translates to real-world settings. LLMs can solve real-world tasks that contain BAPO-easy problems (left plot) with much greater accuracy than BAPO-hard problems (two plots on the right).

Theorems & Definitions (35)

• Definition 1
• Theorem 1
• proof : Proof sketch
• Theorem 2
• proof : Proof sketch
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6
• Definition 2