Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic Reasoning Limits

Xiang Zhang, Juntai Cao, Jiaqi Wei, Yiwei Xu, Chenyu You

TL;DR

Tokenization imposes a fundamental depth constraint on symbolic reasoning in LLMs, constraining the effectiveness of answer-only transformers and CoT-based approaches. The authors formalize token-aware limitations and demonstrate, via theory and extensive experiments, that coarse tokenizations (notably BPE) can cause large degradation in counting and symbolic tasks, while atomically-aligned tokenization restores unit-level fidelity and enables smaller models to match or surpass larger ones. Under ideal CoT assumptions, recurrence can be simulated through external thought tokens, but practical tokenization bottlenecks prevent reliable generalization. The work argues that tokenization design should be treated as a core component of model capability, with significant practical implications for enabling robust symbolic reasoning and efficiency gains.

Abstract

Tokenization is the first - and often underappreciated - layer of computation in language models. While Chain-of-Thought (CoT) prompting enables transformer models to approximate recurrent computation by externalizing intermediate steps, we show that the success of such reasoning is fundamentally bounded by the structure of tokenized inputs. This work presents a theoretical and empirical investigation into how tokenization schemes, particularly subword-based methods like byte-pair encoding (BPE), impede symbolic computation by merging or obscuring atomic reasoning units. We introduce the notion of Token Awareness to formalize how poor token granularity disrupts logical alignment and prevents models from generalizing symbolic procedures. Through systematic evaluation on arithmetic and symbolic tasks, we demonstrate that token structure dramatically affect reasoning performance, causing failure even with CoT, while atomically-aligned formats unlock strong generalization, allowing small models (e.g., GPT-4o-mini) to outperform larger systems (e.g., o1) in structured reasoning. Our findings reveal that symbolic reasoning ability in LLMs is not purely architectural, but deeply conditioned on token-level representations.

Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic Reasoning Limits

TL;DR

Tokenization imposes a fundamental depth constraint on symbolic reasoning in LLMs, constraining the effectiveness of answer-only transformers and CoT-based approaches. The authors formalize token-aware limitations and demonstrate, via theory and extensive experiments, that coarse tokenizations (notably BPE) can cause large degradation in counting and symbolic tasks, while atomically-aligned tokenization restores unit-level fidelity and enables smaller models to match or surpass larger ones. Under ideal CoT assumptions, recurrence can be simulated through external thought tokens, but practical tokenization bottlenecks prevent reliable generalization. The work argues that tokenization design should be treated as a core component of model capability, with significant practical implications for enabling robust symbolic reasoning and efficiency gains.

Abstract

Tokenization is the first - and often underappreciated - layer of computation in language models. While Chain-of-Thought (CoT) prompting enables transformer models to approximate recurrent computation by externalizing intermediate steps, we show that the success of such reasoning is fundamentally bounded by the structure of tokenized inputs. This work presents a theoretical and empirical investigation into how tokenization schemes, particularly subword-based methods like byte-pair encoding (BPE), impede symbolic computation by merging or obscuring atomic reasoning units. We introduce the notion of Token Awareness to formalize how poor token granularity disrupts logical alignment and prevents models from generalizing symbolic procedures. Through systematic evaluation on arithmetic and symbolic tasks, we demonstrate that token structure dramatically affect reasoning performance, causing failure even with CoT, while atomically-aligned formats unlock strong generalization, allowing small models (e.g., GPT-4o-mini) to outperform larger systems (e.g., o1) in structured reasoning. Our findings reveal that symbolic reasoning ability in LLMs is not purely architectural, but deeply conditioned on token-level representations.

Paper Structure

This paper contains 31 sections, 8 equations, 14 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Neural Networks and Arithmetic: A Revisit
  3. Theoretical Limits of Answer-Only Models for Arithmetic and Symbolic Computation.
  4. CoT under Ideal Assumptions Enables General Arithmetic Computation
  5. Inductive Arithmetic Requires Depth
  6. Chain-of-Thought Simulates Recurrent Computation
  7. Tokenization as a Barrier to Chain-of-Thought Computation
  8. Expressiveness and the Token-to-Thought Mapping
  9. Damage Type I: Information Hiding via Token Granularity
  10. Damage Type II: Limited CoT Expressiveness via Token Bottleneck
  11. Formal Failure Case: CoT under BPE Tokenizer
  12. Quantifying Tokenization Effects on Symbolic Computation
  13. Input Design.
  14. Quantifying Degradation.
  15. Experiments
  16. ...and 16 more sections

Figures (14)

  • Figure 1: Illustration of inductive reasoning as performed by humans, RNNs, and LLMs with CoT, respectively.
  • Figure 2: CoT vs Answer Only Generation Models.
  • Figure 3: Four types of string formatting to manipulate tokenization in counting. Examples in the figure are tokenized using the GPT-4o tokenizer. Each string-token type is labeled as (a), (b), (c), and (d) in the diagram. Note that changing the format does not alter the fundamental nature or difficulty of the counting task.
  • Figure 4: Distribution of shifts from the correct count.
  • Figure 5: Counting accuracy (Orange) with respect to target letter frequency (Blue) in Human Natural Language.
  • ...and 9 more figures