Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy

TL;DR

The paper addresses the challenge that large-scale counting in LLMs degrades due to architectural depth limits. It introduces a simple, test-time System-2 strategy that partitions long counting tasks into smaller subproblems using an external separator, counts each subproblem, and aggregates the results, without modifying model parameters. Through behavioral experiments and a mechanistic analysis employing attention studies and causal mediation techniques, the authors show that this approach yields high accuracy on long contexts and reveal how partition-level counts are encoded at partition boundaries, transferred via attention pathways to intermediate reasoning tokens, and finally integrated in later layers. The work advances the interpretability of LLM reasoning under time-saving strategies and suggests a general framework for extending computational capabilities on structured reasoning tasks without training.

Abstract

Large language models (LLMs), despite strong performance on complex mathematical problems, exhibit systematic limitations in counting tasks. This issue arises from architectural limits of transformers, where counting is performed across layers, leading to degraded precision for larger counting problems due to depth constraints. To address this limitation, we propose a simple test-time strategy inspired by System-2 cognitive processes that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. We evaluate this approach using observational and causal mediation analyses to understand the underlying mechanism of this System-2-like strategy. Our mechanistic analysis identifies key components: latent counts are computed and stored in the final item representations of each part, transferred to intermediate steps via dedicated attention heads, and aggregated in the final stage to produce the total count. Experimental results demonstrate that this strategy enables LLMs to surpass architectural limitations and achieve high accuracy on large-scale counting tasks. This work provides mechanistic insight into System-2 counting in LLMs and presents a generalizable approach for improving and understanding their reasoning behavior.

Figures (15)

• Figure 1: System-1 vs. System-2 counting performance as a function of problem size. System-1 performance degrades rapidly and collapses beyond approximately 30 items, reflecting the bounded capacity of the model’s internal counter. In contrast, System-2 counting maintains high accuracy across the entire range by decomposing the task into small solvable sub-problems and aggregating the results.
• Figure 2: Internal mechanism of System-2 test-time counting in LLMs. A large counting task is divided into smaller partitions using an external separator (|). Within each partition, the model performs implicit System-1 counting, where count information accumulates token-by-token and is localized at the final item or separator token (gray blocks). The final count information is transferred via residual streams (green arrows) and stored in the middle-to-late layers (green blocks). These partition-level counts (e.g., 4, 2, 3) are then transferred (orange arrows) through attention pathways to explicit reasoning tokens that report intermediate results (orange blocks). Finally, the intermediate counts are aggregated (purple arrows) to produce the final answer. By keeping each sub-task within the model’s reliable counting range, this System-2 procedure removes the upper bound imposed by the model’s architectural limitations.
• Figure 3: Decoded output probabilities for the unstructured baseline method on Qwen2.5 7B. The heatmap shows the decoded probabilities of model outputs, averaged over different item types, for target counts ranging from 1 to 25. As the count increases beyond 10, the diagonal entries gradually fade, indicating reduced model confidence.
• Figure 4: Attention patterns of selected tokens under the System-2 counting strategy. Attention values are averaged across layers 19 to 23, all heads, and all item types (e.g., different fruit names).
• Figure 5: Average attention of correctly generated intermediate counts to the final item of their partition, alongside attention of the final answer to partition-level counts, across layers. Higher attention values are observed from layer 19 to 23 for both paths.