Counting Ability of Large Language Models and Impact of Tokenization

TL;DR

This work investigates the impact of tokenization on the counting abilities of LLMs, uncovering substantial performance variations based on input tokenization differences, and offers insights into how tokenization choices can undermine models' theoretical computability.

Abstract

Transformers, the backbone of modern large language models (LLMs), face inherent architectural limitations that impede their reasoning capabilities. Unlike recurrent networks, Transformers lack recurrent connections, confining them to constant-depth computation. This restriction places them in the complexity class TC$^0$, making them theoretically incapable of solving tasks that demand increasingly deep reasoning as input length grows. Counting, a fundamental component of many reasoning tasks, also requires reasoning depth to grow linearly to be performed inductively. While previous studies have established the upper limits of counting ability in Transformer-based expert models (i.e., models specifically trained for counting tasks), these findings do not directly extend to general-purpose LLMs due to differences in reasoning mechanisms. Recent work has highlighted how Chain of Thought (CoT) reasoning can help alleviate some of the architectural limitations of Transformers in counting tasks. However, little attention has been paid to the role of tokenization in these models. Unlike expert models that often use character-level tokenization, LLMs typically rely on byte-level (BPE) tokenizers, which fundamentally alters the way reasoning is processed. Our work investigates the impact of tokenization on the counting abilities of LLMs, uncovering substantial performance variations based on input tokenization differences. We provide both theoretical and experimental analyses, offering insights into how tokenization choices can undermine models' theoretical computability, thereby inspiring the design of new tokenization methods to enhance reasoning in LLMs.

Figures (7)

• Figure 1: Experimental results on average counting accuracy based on different tokenization choices, using GPT-4o mini. Our approach treats the model as a black-box, manipulating BPE tokenizers to function differently through carefully engineered string formats.
• Figure 2: Illustration of inductive counting as performed by humans, RNNs, and LLMs with CoT, respectively.
• Figure 3: Four types of string formatting used for counting tasks to manipulate tokenization in LLMs. Examples in the figure are tokenized using the GPT-4o tokenizer. Each string-token type is labeled as (a), (b), (c), and (d) in the diagram. Note that changing the format does not alter the fundamental nature or difficulty of the counting task.
• Figure 4: Distribution of shift from correct count to GPT-calculated count, for each type of string-token fomrat (a), (b), (c) and (d) in order. The statsiticas show the results for letter a at length range [30, 40], as this range the error rate is high. We only calculate the shift when error is made, as correct counting instance does not have any shift.
• Figure 5: Pairwise comparison of counting accuracy for different letters in strings. The left plot shows the distribution of accuracy for a and b in ab strings, with each dot representing the average accuracy for a in a given CoT case (e.g., spaced-string in the [10,20] range), connected to the corresponding accuracy for b in the same setting. The right plot illustrates a similar case for e and z in ez strings. Note: The y-axis limit exceeds [0,1] as the distribution is calculated based on variance and mean, with larger variance pushing the upper bound of the confidence interval beyond the maximum value.