SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding

TL;DR

SubTokenTest targets a practical gap in large language models: precise sub-token understanding obscured by tokenization. The paper presents a ten-task benchmark spanning sequence transformation, text canonicalization, structured data, and 2D spatial reasoning to decouple perceptual from reasoning demands. Key findings show that while reasoning-enabled models reduce sub-token blindness, they incur heavy token costs and display inverted-U performance trends with increasing thinking budgets; smaller models struggle more and may degrade with extended traces. Probing experiments reveal that character-level information is encoded across layers in a form-dependent manner, offering guidance for designing prompts and architectures that better preserve sub-token fidelity. Overall, SubTokenTest provides a rigorous, real-world-oriented framework to diagnose and quantify sub-token understanding and its practical implications for downstream tasks.

Abstract

Recent advancements in large language models (LLMs) have significantly enhanced their reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-token understanding. In this regard, we introduce SubTokenTest, a comprehensive benchmark that assesses sub-token understanding through practical, utility-driven tasks. Our benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-token reasoning and explore how character-level information is encoded within the hidden states.

Figures (31)

• Figure 1: Example cases where tokenization may cause confusion in practical tasks.
• Figure 2: SubTokenTest categories.
• Figure 3: The two subfigures illustrate distinct error patterns: (a) tokenization-induced errors in DeepSeek-V3 (highlighted in red), and (b) overthinking errors in DeepSeek-R1 (highlighted in purple).
• Figure 4: The effects of the number of thinking tokens on the task performance. Here we evaluate DS-distill-Qwen-2.5-7B on Biological Sequence Manipulation with the metric of normalized similarity.
• Figure 5: The Macro F1 results of linear probing the last token of certain token sequences. The dot line is the experimental group, and the square line is the corresponding baseline trained with shuffled labels.