Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-word Tokenization for Sequence Compression

Leonidas Gee, Leonardo Rigutini, Marco Ernandes, Andrea Zugarini

TL;DR

The paper tackles the high computational cost of large language models by compressing input sequences through a Multi-Word Tokenizer that encodes frequent multi-word expressions as single tokens. It introduces a greedy n-gram selection process to augment the tokenizer vocabulary and applies Fast Vocabulary Transfer to initialize embeddings for new tokens, enabling efficient integration. Across medical, legal, and patent domains, MWT delivers substantial sequence compression and speedups while maintaining or slightly improving performance, and remains effective when combined with DistilBERT and other compression techniques. The work demonstrates a practical path to faster inference in domain-specific settings and invites future work on broader models and generation tasks.

Abstract

Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this paper, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.

Multi-word Tokenization for Sequence Compression

TL;DR

The paper tackles the high computational cost of large language models by compressing input sequences through a Multi-Word Tokenizer that encodes frequent multi-word expressions as single tokens. It introduces a greedy n-gram selection process to augment the tokenizer vocabulary and applies Fast Vocabulary Transfer to initialize embeddings for new tokens, enabling efficient integration. Across medical, legal, and patent domains, MWT delivers substantial sequence compression and speedups while maintaining or slightly improving performance, and remains effective when combined with DistilBERT and other compression techniques. The work demonstrates a practical path to faster inference in domain-specific settings and invites future work on broader models and generation tasks.

Abstract

Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this paper, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.
Paper Structure (24 sections, 4 figures, 5 tables)

This paper contains 24 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Sequence Compression.
  4. Multi-word Tokenizer
  5. N-gram Selection.
  6. Fast Vocabulary Transfer.
  7. Experiments
  8. Experimental Setup
  9. Implementation Details.
  10. Choice of N.
  11. Datasets
  12. ADE.
  13. LEDGAR.
  14. PATENT.
  15. Results
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Tokenization using generic $\mathcal{T}_{gen}$ and adapted $\mathcal{T}_{100}$ tokenizers. $\mathcal{T}_{gen}^{1000}$ and $\mathcal{T}_{100}^{1000}$ are extended with the top-1000 bigrams. Tokens obtained with domain-adaptation or MWT are highlighted in orange and blue respectively. MWTs are shown to be highly complementary to existing tokenizers for sequence compression.
  • Figure 2: Sketch of the Multi-word Tokenizer pipeline. First, n-grams are statistically learned from the training set. Then, the top-K n-grams are added to the vocabulary of the tokenizer. N-grams are merged from left to right within a sequence after pre-tokenization.
  • Figure 3: Fast Vocabulary Transfer. The pre-trained embeddings of existing tokens are combined to form the embeddings of the newly adapted vocabulary.
  • Figure 4: Plot of macro-F1 against maximum sequence length. The generic $\mathcal{T}_{gen}$ and adapted $\mathcal{T}_{100}$ tokenizers are represented by solid and dashed lines respectively. MWTs are shown to be more robust on shorter sequence lengths, thus allowing for major speedups via early sequence truncation.