Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lossless Vocabulary Reduction for Auto-Regressive Language Models

Daiki Chijiwa, Taku Hasegawa, Kyosuke Nishida, Shin'ya Yamaguchi, Tomoya Ohba, Tamao Sakao, Susumu Takeuchi

TL;DR

This work addresses the challenge of cooperative inference across auto regressive language models trained with different vocabularies. It introduces a formal lossless vocabulary reduction framework that maps next token distributions from a full vocabulary to any sub vocabulary through nested tokenization while preserving the underlying text distribution. A key contribution is the relative covering based computation and efficient algorithms that enable practical inference time reductions. As an application, the maximal common vocabulary enables ensemble across models with different tokenizers, achieving comparable accuracy to byte level ensembles with improved efficiency. Empirical results demonstrate near lossless reductions and effective cross model cooperation, highlighting practical benefits for multi tokenizer ecosystems.

Abstract

Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularly, auto-regressive language models generate texts token by token, i.e., by predicting the next-token distribution given the previous ones, and thus tokenization directly affects their efficiency in text generation. Since each language model has their own vocabulary as a set of possible tokens, they struggle to cooperate with each other at the level of next-token distributions such as model ensemble. In this paper, we establish a theoretical framework of lossless vocabulary reduction, which efficiently converts a given auto-regressive language model into the one with an arbitrarily small vocabulary without any loss in accuracy. As an application, we demonstrate that language models with different tokenization can cooperate with each other efficiently through their maximal common vocabulary.

Lossless Vocabulary Reduction for Auto-Regressive Language Models

TL;DR

This work addresses the challenge of cooperative inference across auto regressive language models trained with different vocabularies. It introduces a formal lossless vocabulary reduction framework that maps next token distributions from a full vocabulary to any sub vocabulary through nested tokenization while preserving the underlying text distribution. A key contribution is the relative covering based computation and efficient algorithms that enable practical inference time reductions. As an application, the maximal common vocabulary enables ensemble across models with different tokenizers, achieving comparable accuracy to byte level ensembles with improved efficiency. Empirical results demonstrate near lossless reductions and effective cross model cooperation, highlighting practical benefits for multi tokenizer ecosystems.

Abstract

Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularly, auto-regressive language models generate texts token by token, i.e., by predicting the next-token distribution given the previous ones, and thus tokenization directly affects their efficiency in text generation. Since each language model has their own vocabulary as a set of possible tokens, they struggle to cooperate with each other at the level of next-token distributions such as model ensemble. In this paper, we establish a theoretical framework of lossless vocabulary reduction, which efficiently converts a given auto-regressive language model into the one with an arbitrarily small vocabulary without any loss in accuracy. As an application, we demonstrate that language models with different tokenization can cooperate with each other efficiently through their maximal common vocabulary.

Paper Structure

This paper contains 41 sections, 7 theorems, 37 equations, 3 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Formulation of Tokens
  4. Texts.
  5. Tokenization.
  6. Formulation and Properties of Language Models
  7. Language model over tokens.
  8. Next-token distribution.
  9. Valid tokens and minimal covering.
  10. Lossless Vocabulary Reduction
  11. Formulation of Vocabulary Reduction
  12. Nested tokenization.
  13. Vocabulary reduction.
  14. Computation of the Next-Token Distribution
  15. An illustrative example.
  16. ...and 26 more sections

Key Result

Lemma 2.1

Let $\mathcal{V}$ be any vocabulary and $p_\mathcal{V}$ be the token distribution associated with the text distribution $p_\mathrm{text}$. For any string $a_1\cdots a_N \in \mathcal{A}^N$, we have

Figures (3)

  • Figure 1: Overview of lossless vocabulary reduction. Instead of sampling tokens from the original next-token distribution over $\mathcal{V}$, we can inductively compute and sample from the equivalent distribution over the sub-vocabulary $\mathcal{V}_\mathrm{sub}$ while keeping its accuracy. See \ref{['sec:lossless vocabulary reduction']} for notations and details.
  • Figure 2: An example of text generation by greedy decoding of the original model (Llama3.2-3B) and the vocabulary-reduced models with varying maximal token lengths from $1$ to $8$ bytes. Each token is colored periodically for visibility. See \ref{['app:sec:additional examples of vocabulary reduction']} for other examples.
  • Figure : Lossless Vocabulary Reduction (LVR).

Theorems & Definitions (12)

  • Lemma 2.1: phan2025exactvieira2025language
  • Theorem 3.1
  • Lemma 3.2
  • proof
  • Lemma 3.3
  • proof
  • Theorem 3.4
  • proof
  • Lemma A.1
  • proof
  • ...and 2 more