Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging

Azam Nouri

Abstract

Subword tokenization is a key design choice for modern language models, including large language models (LLMs), with byte- and character-level BPE serving as a widely used baseline. Standard BPE selects merges by raw pair frequency, which favors compression but can conflate true adjacency cohesion with pairs that are frequent due to high marginal counts. This paper introduces Significance-Gain BPE, a drop-in alternative merge criterion that measures cohesion via a z-statistic under an independence null model and combines it with an explicit compression-aware gain term. Significance-Gain BPE is evaluated on WikiText-103 (raw) character slices using a small causal Transformer language model, reporting both token-dependent perplexity and the tokenizer-invariant metric bits per character (BPC). At a representative operating point, Significance-Gain BPE reduces validation and test perplexity by 13% and 12%, respectively, and improves validation and test BPC by about 0.9 to 1.0%. A vocabulary-size sweep further shows lower BPC in most closest-compression comparisons, suggesting that statistically grounded merge selection can improve predictive efficiency per unit of raw text across a range of compression regimes.

Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging

Abstract

Subword tokenization is a key design choice for modern language models, including large language models (LLMs), with byte- and character-level BPE serving as a widely used baseline. Standard BPE selects merges by raw pair frequency, which favors compression but can conflate true adjacency cohesion with pairs that are frequent due to high marginal counts. This paper introduces Significance-Gain BPE, a drop-in alternative merge criterion that measures cohesion via a z-statistic under an independence null model and combines it with an explicit compression-aware gain term. Significance-Gain BPE is evaluated on WikiText-103 (raw) character slices using a small causal Transformer language model, reporting both token-dependent perplexity and the tokenizer-invariant metric bits per character (BPC). At a representative operating point, Significance-Gain BPE reduces validation and test perplexity by 13% and 12%, respectively, and improves validation and test BPC by about 0.9 to 1.0%. A vocabulary-size sweep further shows lower BPC in most closest-compression comparisons, suggesting that statistically grounded merge selection can improve predictive efficiency per unit of raw text across a range of compression regimes.
Paper Structure (38 sections, 15 equations, 1 figure, 2 tables, 1 algorithm)

This paper contains 38 sections, 15 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation.
  3. Core idea: cohesion $\times$ gain.
  4. Evaluation: token-dependent vs. token-invariant metrics.
  5. Contributions.
  6. Paper organization.
  7. Significance-Gain BPE vs. Standard Frequency BPE
  8. Standard BPE as a compression-driven greedy algorithm
  9. Limitation (marginals vs. cohesion).
  10. Independence null model and statistical cohesion
  11. Connection to PMI (intuition).
  12. Significance-Gain score used in the implementation
  13. Counts and notation.
  14. Cohesion (significance) term.
  15. Compression (gain) term.
  16. ...and 23 more sections

Figures (1)

  • Figure 1: Validation bits-per-character (BPC) versus tokens-per-character (TPC) across a vocabulary-size sweep. Lower is better on both axes. Significance-Gain BPE (alpha=0.25) is typically below frequency-BPE, indicating improved predictive efficiency per unit of text across most compression regimes.