Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Say Anything but This: When Tokenizer Betrays Reasoning in LLMs

Navid Ayoobi, Marcus I Armstrong, Arjun Mukherjee

TL;DR

The paper identifies tokenizer non-uniqueness as a hidden source of reasoning fragility in LLMs, proposing a tokenization-consistency probe that reveals phantom edits where token-ID changes do not alter surface text. Through about 11k trials across open-source models, it catalogs eight artifact classes caused by non-injective token-to-text mappings and demonstrates that a post-hoc token-ID masking can substantially reduce such errors, revealing latent reasoning capabilities. The findings show that scaling alone does not fix tokenization-induced misalignments, and motivate tokenizer-aware training or redesign to ensure robust surface-level reasoning. The work highlights a practical pathway to improve reliability of LLM reasoning without solely increasing model size or data, by addressing the foundational representation layer.

Abstract

Large language models (LLMs) reason over discrete token ID sequences, yet modern subword tokenizers routinely produce non-unique encodings: multiple token ID sequences can detokenize to identical surface strings. This representational mismatch creates an unmeasured fragility wherein reasoning processes can fail. LLMs may treat two internal representations as distinct "words" even when they are semantically identical at the text level. In this work, we show that tokenization can betray LLM reasoning through one-to-many token ID mappings. We introduce a tokenization-consistency probe that requires models to replace designated target words in context while leaving all other content unchanged. The task is intentionally simple at the surface level, enabling us to attribute failures to tokenizer-detokenizer artifacts rather than to knowledge gaps or parameter limitations. Through analysis of over 11000 replacement trials across state-of-the-art open-source LLMs, we find a non-trivial rate of outputs exhibit phantom edits: cases where models operate under the illusion of correct reasoning, a phenomenon arising from tokenizer-induced representational defects. We further analyze these cases and provide a taxonomy of eight systematic tokenizer artifacts, including whitespace-boundary shifts and intra-word resegmentation. These findings indicate that part of apparent reasoning deficiency originates in the tokenizer layer, motivating tokenizer-level remedies before incurring the cost of training ever-larger models on ever-larger corpora.

Say Anything but This: When Tokenizer Betrays Reasoning in LLMs

TL;DR

The paper identifies tokenizer non-uniqueness as a hidden source of reasoning fragility in LLMs, proposing a tokenization-consistency probe that reveals phantom edits where token-ID changes do not alter surface text. Through about 11k trials across open-source models, it catalogs eight artifact classes caused by non-injective token-to-text mappings and demonstrates that a post-hoc token-ID masking can substantially reduce such errors, revealing latent reasoning capabilities. The findings show that scaling alone does not fix tokenization-induced misalignments, and motivate tokenizer-aware training or redesign to ensure robust surface-level reasoning. The work highlights a practical pathway to improve reliability of LLM reasoning without solely increasing model size or data, by addressing the foundational representation layer.

Abstract

Large language models (LLMs) reason over discrete token ID sequences, yet modern subword tokenizers routinely produce non-unique encodings: multiple token ID sequences can detokenize to identical surface strings. This representational mismatch creates an unmeasured fragility wherein reasoning processes can fail. LLMs may treat two internal representations as distinct "words" even when they are semantically identical at the text level. In this work, we show that tokenization can betray LLM reasoning through one-to-many token ID mappings. We introduce a tokenization-consistency probe that requires models to replace designated target words in context while leaving all other content unchanged. The task is intentionally simple at the surface level, enabling us to attribute failures to tokenizer-detokenizer artifacts rather than to knowledge gaps or parameter limitations. Through analysis of over 11000 replacement trials across state-of-the-art open-source LLMs, we find a non-trivial rate of outputs exhibit phantom edits: cases where models operate under the illusion of correct reasoning, a phenomenon arising from tokenizer-induced representational defects. We further analyze these cases and provide a taxonomy of eight systematic tokenizer artifacts, including whitespace-boundary shifts and intra-word resegmentation. These findings indicate that part of apparent reasoning deficiency originates in the tokenizer layer, motivating tokenizer-level remedies before incurring the cost of training ever-larger models on ever-larger corpora.
Paper Structure (22 sections, 4 figures)

This paper contains 22 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Tokenization in modern LLMs
  4. Advances in LLM reasoning
  5. Tokenization as a reasoning bottleneck
  6. Method
  7. Task setup overview
  8. Experimental setup
  9. Result and discussion
  10. Model size is not always the bottleneck for reasoning
  11. Tokenizer-induced phantom edits
  12. Error 1: whitespace-boundary shift
  13. Error 2: whitespace detachment/reattachment
  14. Error 3: newline/whitespace substitution
  15. Error 4: intra-word resegmentation
  16. ...and 7 more sections

Figures (4)

  • Figure 1: The overview of the proposed tokenization-consistency probe.
  • Figure 2: Distribution of outcomes across LLM families and parameter scales for the word-replacement task. The "Different" category (red) highlights tokenizer-induced phantom edits, which persist across all model sizes and families.
  • Figure 3: Heatmap of subtoken fragmentation transitions for "Different" class instances using the Gemma3-4B model. The x and y axes denote the number of token fragments in the input and output, respectively. Cell values indicate the frequency of each transition, while the color scale represents the mean character length of the words.
  • Figure 4: Distribution of outcomes across LLM families after applying the token-ID masking intervention.