Training Language Models with homotokens Leads to Delayed Overfitting

TL;DR

The paper tackles tokenization-induced non-uniqueness in language model training by introducing homotokens—meaning-preserving, alternate subword segmentations of the same lexical item. It presents a lightweight two-branch architecture with a causal encoder and block-causal cross-attention to condition next-token prediction on homotoken variants while preserving the canonical token interface and objective. Empirically, pretraining with homotokens delays overfitting and improves data efficiency across various tasks, with gains in multilingual settings depending on tokenizer quality. The approach offers a practical, modular path to tokenization invariance that complements existing regularization techniques and avoids costly tokenizer redesigns. Overall, homotokens demonstrate a new axis for augmentation at the computational-linguistic interface, enabling more robust and data-efficient LM training."

Abstract

Subword tokenization introduces a computational layer in language models where many distinct token sequences decode to the same surface form and preserve meaning, yet induce different internal computations. Despite this non-uniqueness, language models are typically trained using a single canonical longest-prefix tokenization. We formalize homotokens-alternative valid subword segmentations of the same lexical item-as a strictly meaning-preserving form of data augmentation. We introduce a lightweight training architecture that conditions canonical next-token prediction on sampled homotoken variants via an auxiliary causal encoder and block-causal cross-attention, without modifying the training objective or token interface. In data-constrained pretraining, homotoken augmentation consistently delays overfitting under repeated data exposure and improves generalization across diverse evaluation datasets. In multilingual fine-tuning, we find that the effectiveness of homotokens depends on tokenizer quality: gains are strongest when canonical tokens are highly compressed and diminish when the tokenizer already over-fragments the input. Overall, homotokens provide a simple and modular mechanism for inducing tokenization invariance in language models.

Figures (6)

• Figure 1: Diagram of our architecture. The main causal decoder operates on canonical longest-prefix BPE tokens, while a lightweight single-block causal encoder consumes homotoken segmentations (e.g., dinosaur$\rightarrow$dino + saur | d + inosaur) and injects them into the main trunk via a block-causal cross-attention operation.
• Figure 2: Evaluations on various chat datasets across pretraining duration for two model sizes: $\mu=1$ - top and $\mu=2$ - bottom. We show training runs with the number of data repetitions $R \in \{16, 24, 32, 64\}$ for clarity. Training with homotokens consistently delays overfitting in these scenarios.
• Figure 3: Pretraining curves for various numbers of data repetitions for an 88M parameter model ($\mu = 1$, left) and a 244M parameter model ($\mu = 2$, right). Training with homotokens delays overfitting for $R > 16$.
• Figure 4: Loss during pretraining for runs using attention dropout and token-level Gaussian noise as augmentations for $R = 32$ and $\mu =1$.
• Figure 5: Loss across two benchmarks for runs using attention dropout (top) and token-level Gaussian noise (bottom) as augmentations for $R = 32$ and $\mu = 1$.

Theorems & Definitions (4)

• Definition 3.1: Text Augmentation
• Definition 3.2: Canonical and Non-Canonical Tokenizations
• Definition 3.3: Homotoken
• proof