Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar

Yinxi Li, Yuntian Deng, Pengyu Nie

TL;DR

TokDrift exposes a misalignment between subword LLM tokenization and programming language grammar by applying semantic-preserving rewrites and measuring cross-model sensitivity across bug fixing, summarization, and translation tasks. The framework reveals that minor formatting changes can shift outputs, with root causes traced to early embeddings and token boundary misalignments, and larger models generally exhibiting reduced sensitivity. The work provides a large-scale, open framework and dataset for evaluating grammar-aware tokenization approaches, and argues for developing grammar-aligned tokenizers to improve code reliability and robustness in LLMs. Overall, TokDrift highlights tokenization as a critical design factor in code LLMs and motivates grammar-aware tokenization as a pathway to more robust code understanding and generation.

Abstract

Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result, semantically identical code snippets can be tokenized differently depending on superficial factors such as whitespace or identifier naming. To measure the impact of this misalignment, we introduce TokDrift, a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. Across nine code LLMs, including large ones with over 30B parameters, even minor formatting changes can cause substantial shifts in model behavior. Layer-wise analysis shows that the issue originates in early embeddings, where subword segmentation fails to capture grammar token boundaries. Our findings identify misaligned tokenization as a hidden obstacle to reliable code understanding and generation, highlighting the need for grammar-aware tokenization for future code LLMs.

TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar

TL;DR

TokDrift exposes a misalignment between subword LLM tokenization and programming language grammar by applying semantic-preserving rewrites and measuring cross-model sensitivity across bug fixing, summarization, and translation tasks. The framework reveals that minor formatting changes can shift outputs, with root causes traced to early embeddings and token boundary misalignments, and larger models generally exhibiting reduced sensitivity. The work provides a large-scale, open framework and dataset for evaluating grammar-aware tokenization approaches, and argues for developing grammar-aligned tokenizers to improve code reliability and robustness in LLMs. Overall, TokDrift highlights tokenization as a critical design factor in code LLMs and motivates grammar-aware tokenization as a pathway to more robust code understanding and generation.

Abstract

Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result, semantically identical code snippets can be tokenized differently depending on superficial factors such as whitespace or identifier naming. To measure the impact of this misalignment, we introduce TokDrift, a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. Across nine code LLMs, including large ones with over 30B parameters, even minor formatting changes can cause substantial shifts in model behavior. Layer-wise analysis shows that the issue originates in early embeddings, where subword segmentation fails to capture grammar token boundaries. Our findings identify misaligned tokenization as a hidden obstacle to reliable code understanding and generation, highlighting the need for grammar-aware tokenization for future code LLMs.
Paper Structure (29 sections, 4 equations, 22 figures, 9 tables)

This paper contains 29 sections, 4 equations, 22 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Tokenization
  4. PL Tokenization
  5. TokDrift Framework
  6. Benchmarks
  7. Models
  8. Rewrite Rules
  9. Metrics
  10. Evaluation
  11. Results
  12. Impact of Model Size
  13. Impact of Identifier Fragment Changes
  14. Root Cause Analyses
  15. Word Frequency Analysis
  16. ...and 14 more sections

Figures (22)

  • Figure 1: Workflow of TokDrift, our framework for quantifying LLM sensitivity to semantic-preserving code rewrite rules.
  • Figure 2: Example of tokenization misalignment. Adding a space between dot (".") and "factorial" causes a significant change in token sequences, from [".factor", "ial"] to [".", "␣ factorial"]. Consequently, the LLM's code translation prediction shifts from incorrect (naming the factorial function as "comb" and later referring to it as "combin") to correct.
  • Figure 3: TokDrift workflow and example.
  • Figure 4: Heatmap of (code) LLMs' vocabulary distances AmbaHombaiahETAL21Dynamic.
  • Figure 5: grouped by naming rewrite rule
  • ...and 17 more figures