Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LBPE: Long-token-first Tokenization to Improve Large Language Models

Haoran Lian, Yizhe Xiong, Zijia Lin, Jianwei Niu, Shasha Mo, Hui Chen, Peng Liu, Guiguang Ding

TL;DR

LBPE generates tokens according to their descending order of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process, which smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance.

Abstract

The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its success, a critical challenge persists: long tokens, rich in semantic information, have fewer occurrences in tokenized datasets compared to short tokens, which can result in imbalanced learning issue across different tokens. To address that, we propose LBPE, which prioritizes long tokens during the encoding process. LBPE generates tokens according to their reverse ranks of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process. Consequently, LBPE smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance. Extensive experiments across diverse language modeling tasks demonstrate that LBPE consistently outperforms the original BPE, well demonstrating its effectiveness.

LBPE: Long-token-first Tokenization to Improve Large Language Models

TL;DR

LBPE generates tokens according to their descending order of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process, which smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance.

Abstract

The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its success, a critical challenge persists: long tokens, rich in semantic information, have fewer occurrences in tokenized datasets compared to short tokens, which can result in imbalanced learning issue across different tokens. To address that, we propose LBPE, which prioritizes long tokens during the encoding process. LBPE generates tokens according to their reverse ranks of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process. Consequently, LBPE smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance. Extensive experiments across diverse language modeling tasks demonstrate that LBPE consistently outperforms the original BPE, well demonstrating its effectiveness.

Paper Structure

This paper contains 18 sections, 2 equations, 1 figure, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments
  4. Experimental Setup
  5. Datasets
  6. Tokenizer
  7. Model
  8. Training
  9. Experimental Results
  10. Discussion
  11. Token Length Distribution
  12. Never too Late to Use LBPE
  13. Various Vocabulary Sizes
  14. More Training Tokens
  15. Higher Compression Rate
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Compared to the original BPE that iteratively merges token pairs according to the rank in vocabulary, LBPE merges a sequence of unit tokens into the final token representation directly according to the reverse rank of token length. Then LBPE can encode "_Capital" as one token which is semantically more accurate. However, the original BPE merges "al" and "s" first according to the rank in vocabulary, resulting in the failure to derive "_Capital" in the end. "_" represents space character.