Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling LLM Pre-training with Vocabulary Curriculum

Fangyuan Yu

TL;DR

This work introduces vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size, and alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities.

Abstract

Modern language models rely on static vocabularies, fixed before pretraining, in contrast to the adaptive vocabulary acquisition observed in human language learning. To bridge this gap, we introduce vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size. Our method alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities. This approach naturally gives rise to an optimal computation allocation pattern: longer tokens capture predictable content, while shorter tokens focus on more complex, harder-to-predict contexts. Experiments on small-scale GPT models demonstrate improved scaling efficiency, reinforcing the effectiveness of dynamic tokenization. We release our code to support further research and plan to extend our experiments to larger models and diverse domains.

Scaling LLM Pre-training with Vocabulary Curriculum

TL;DR

This work introduces vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size, and alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities.

Abstract

Modern language models rely on static vocabularies, fixed before pretraining, in contrast to the adaptive vocabulary acquisition observed in human language learning. To bridge this gap, we introduce vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size. Our method alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities. This approach naturally gives rise to an optimal computation allocation pattern: longer tokens capture predictable content, while shorter tokens focus on more complex, harder-to-predict contexts. Experiments on small-scale GPT models demonstrate improved scaling efficiency, reinforcing the effectiveness of dynamic tokenization. We release our code to support further research and plan to extend our experiments to larger models and diverse domains.

Paper Structure

This paper contains 13 sections, 5 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Relevant Work
  3. Tokenization Methods and Limitations
  4. Curriculum Learning
  5. Entropy Aware Tokenization
  6. Approach
  7. Entropy-Guided Vocabulary Update
  8. Vocabulary Curriculum Learning
  9. Experiments
  10. Experimental Setup
  11. Incremental Vocabulary Curriculum
  12. Analysis of Improvement Mechanisms
  13. Implication and Future work

Figures (6)

  • Figure 1: Scaling better with vocabulary curriculum
  • Figure 2: Scaling better with vocabulary curriculum
  • Figure 3: Token grouping process based on entropy patterns from a trained character-level language model
  • Figure 4: Incremental vocabulary learning shows noticeable improvement which scales with vocabulary size in log-linear fashion
  • Figure 5: Longer tokens has smaller BPC, contributing to smaller global BPC
  • ...and 1 more figures