Semantic Content Determines Algorithmic Performance

TL;DR

WhatCounts investigates whether algorithmic primitives in LLMs are truly invariant to the semantic class of their inputs. The authors introduce WhatCounts, an atomic counting benchmark using unambiguous, delimited lists to isolate semantic content from tokenization and multi-step reasoning, and quantify the semantic gap with $\Delta_{\mathrm{sem}}(m) = \max_{e \in \mathcal{E}} \mathrm{Acc}(m,e) - \min_{e \in \mathcal{E}} \mathrm{Acc}(m,e)$. They demonstrate that frontier LLMs exhibit over 40 percentage points variation across semantic classes under identical conditions, and that surface ablations (tokenization, separators, or explicit structure) do not eliminate the gap. Training data and fine-tuning produce unstable, dataset-dependent shifts in both accuracy and the semantic gap, indicating that LLMs approximate algorithms in a content-dependent way. The findings imply practical risks for production and agentic pipelines where content-conditioned behavior can cascade through tool use and decision making, challenging the premise of prompt programming as a robust solution. WhatCounts highlights the need for semantic-aware models and benchmarks to ensure reliable, invariant computation in real-world AI systems.

Abstract

Counting should not depend on what is being counted; more generally, any algorithm's behavior should be invariant to the semantic content of its arguments. We introduce WhatCounts to test this property in isolation. Unlike prior work that conflates semantic sensitivity with reasoning complexity or prompt variation, WhatCounts is atomic: count items in an unambiguous, delimited list with no duplicates, distractors, or reasoning steps for different semantic types. Frontier LLMs show over 40% accuracy variation depending solely on what is being counted - cities versus chemicals, names versus symbols. Controlled ablations rule out confounds. The gap is semantic, and it shifts unpredictably with small amounts of unrelated fine-tuning. LLMs do not implement algorithms; they approximate them, and the approximation is argument-dependent. As we show with an agentic example, this has implications beyond counting: any LLM function may carry hidden dependencies on the meaning of its inputs.

Figures (19)

• Figure 1: Overview of WhatCounts. The top panel contrasts the theoretical ideal with the observed behavior of LLMs. In theory, an algorithm (e.g., counting) should be invariant to the meaning of its arguments, performing identically for lists of cities or emojis. Results indicate that accuracy varies strongly across semantic categories. We further run controlled ablations (e.g., token shuffling, token-count controls, and explicit separator specification), none of which eliminate the semantic class dependence, effectively demonstrating the inherent semantic fragility.
• Figure 2: The semantic gap varies significantly across models, with top performers showing the highest variance between minimum and maximum semantic class scores. The horizontal bars represent the semantic gap (the smaller, the better), with the classes at the ends of the bars representing the minimum and maximum scores for each model. \ref{['fig:general_app']} shows the detailed results for each semantic class. We find that the better-performing models are also more susceptible to changes in semantic class.
• Figure 3: Token-controlled lists ablation reveals that semantic gaps are wider when fixing tokens compared to fixing classes counts. The bars represent the semantic gap difference with respect to the result of the list count being fixed. Negative results indicate that the semantic gap was reduced when fixing the token count instead of item count. We observe that the semantic gap generally increases when fixing the token counts across classes in the list, with the notable exception of o3.
• Figure 4: Most models demonstrate superior performance in identifying and wrapping items of the different classes compared to basic counting. The results show that, models, except for o3, understand semantic classes better than they can count them.
• Figure 5: The semantics that the tokens carry have some effect as there are substantial differences with respect to the non-shuffled case. Absolute difference of shuffling the tokens, and not doing it. Note that Claude is not present in this figure because we cannot run the experiment without access to the tokenizer.