Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE

Rares Dolga, Lucas Maystre, Tudor Berariu, David Barber

TL;DR

This paper proposes a dynamic character grouping method that leverages the structure of existing BPE tokenization without requiring additional models and offers efficient, flexible, and language-agnostic representations.

Abstract

Subword tokenization methods like Byte Pair Encoding (BPE) are widely used in large language models due to their balance of vocabulary compactness and representational power. However, they suffer from inefficiencies in representing rare words and require large embedding matrices. Character-level models address these issues but introduce performance bottlenecks, particularly in Transformer-based architectures. Recent hierarchical models attempt to merge the benefits of both paradigms by grouping characters into patches, but existing patching strategies either rely on whitespace-limiting applicability to certain languages, or require auxiliary models that introduce new dependencies. In this paper, we propose a dynamic character grouping method that leverages the structure of existing BPE tokenization without requiring additional models. By appending explicit end-of-patch markers to BPE tokens and introducing a second-level BPE compression stage to control patch granularity, our method offers efficient, flexible, and language-agnostic representations. Empirical results demonstrate that our approach matches or exceeds the performance of dynamic entropy- and whitespace-based patching strategies, while maintaining a compact vocabulary.

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE

TL;DR

This paper proposes a dynamic character grouping method that leverages the structure of existing BPE tokenization without requiring additional models and offers efficient, flexible, and language-agnostic representations.

Abstract

Subword tokenization methods like Byte Pair Encoding (BPE) are widely used in large language models due to their balance of vocabulary compactness and representational power. However, they suffer from inefficiencies in representing rare words and require large embedding matrices. Character-level models address these issues but introduce performance bottlenecks, particularly in Transformer-based architectures. Recent hierarchical models attempt to merge the benefits of both paradigms by grouping characters into patches, but existing patching strategies either rely on whitespace-limiting applicability to certain languages, or require auxiliary models that introduce new dependencies. In this paper, we propose a dynamic character grouping method that leverages the structure of existing BPE tokenization without requiring additional models. By appending explicit end-of-patch markers to BPE tokens and introducing a second-level BPE compression stage to control patch granularity, our method offers efficient, flexible, and language-agnostic representations. Empirical results demonstrate that our approach matches or exceeds the performance of dynamic entropy- and whitespace-based patching strategies, while maintaining a compact vocabulary.
Paper Structure (15 sections, 7 equations, 8 figures, 4 tables, 1 algorithm)

This paper contains 15 sections, 7 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Hierarchical BPE
  4. Hierarchical Model
  5. Metrics
  6. Experiments
  7. Experimental Setup
  8. Baseline Comparison
  9. Increasing the training time.
  10. Evaluation on Chinese language
  11. Increasing the vocabulary
  12. Ablation on S
  13. Downstream Tasks
  14. Related Work
  15. Conclusion

Figures (8)

  • Figure 1: Hierarchical Model: Next patch prediction with autoregressive character prediction per patch. A patch is given by a BPE token."<!>" represent end of token.
  • Figure 2: Hierarchical Tokenisation at test time.
  • Figure 3: Histogram of lengths for all tokens in the GPT2 tokenizer. The first plot is truncated on the x-axis, from maximum token length $M=93$.
  • Figure 4: Average Patch Lengths for SlimPajama dataset. "*" represents the unbounded entropy model.
  • Figure 5: Test performance for different amounts of training.
  • ...and 3 more figures