From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks

Abstract

Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks. Although this limitation has been noted, the internal reasons remain unclear. We use character counting (e.g., "How many p's are in apple?") as a minimal, controlled probe that isolates token-level reasoning from higher-level confounds. Using this setting, we uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma: models often compute the correct answer internally yet fail to express it at the output layer. Through mechanistic analysis combining probing classifiers, activation patching, logit lens analysis, and attention head tracing, we show that character-level information is encoded in early and mid-layer representations. However, this information is attenuated by a small set of components in later layers, especially the penultimate and final layer MLP. We identify these components as negative circuits: subnetworks that downweight correct signals in favor of higher-probability but incorrect outputs. Our results lead to two contributions. First, we show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph. This explains why such errors persist and can worsen under scaling and instruction tuning. Second, we provide evidence that LLM forward passes implement a form of competitive decoding, in which correct and incorrect hypotheses coexist and are dynamically reweighted, with final outputs determined by suppression as much as by amplification. These findings carry implications for interpretability and robustness: simple symbolic reasoning exposes weaknesses in modern LLMs, underscoring need for design strategies that ensure information is encoded and reliably used.

Figures (16)

• Figure 1: Overview of how character-level information flows through an LLM during the task.
• Figure 2: Accuracy of predicted letter counts. The dotted line indicates the random baseline accuracy. We calculate the proportion of samples where deviation is zero
• Figure 3: Layerwise probe accuracy
• Figure 4: Logit difference across layers for the correct count token, averaged over approximately 100 samples. Higher values indicate layers where suppression of the correct answer occurs. The final label final denotes the final hidden state after all layers. Results are computed over samples where the model produces an incorrect prediction, corresponding to the "always predict 1" strategy for Qwen2.5-3B and the near-uniform random strategy for LLaMA3.2-3B.
• Figure 5: Activation patching heatmaps when corrupting the word, letter or both letter and word for Qwen2.5-3B attention heads averaged over 1000 samples. Green regions indicate components where clean activations restore the correct behavior, helping identify localized circuits for character counting. The Red regions indicate components which diminish performance when patched in.