Peek2: A Regex-free implementation of pretokenizers for Byte-level BPE

TL;DR

The paper tackles inefficiencies and safety concerns in Regex-based pretokens for Byte-level BPE tokenization by introducing Peek2, a Regex-free, CPU-based pretokenizer that preserves the original presegmentation while achieving $O(n)$ time. It is designed as a drop-in replacement for cl100k-like pretokens used in models such as GPT-3, LLaMa-3, and Qwen-2.5, and demonstrates bug-for-bug compatibility with the Regex baseline. Empirical results show throughput improvements up to $1.11×$ across representative tasks, underscoring practical benefits for preprocessing and training pipelines in large language models. The work emphasizes safety, determinism, and performance, and outlines avenues for future GPU acceleration and broader pretokenizer compatibility.

Abstract

Pretokenization is a crucial, sequential pass in Byte-level BPE tokenizers. Our proposed new implementation, Peek2, serves as a drop-in replacement for cl100k-like pretokenizers used in GPT-3, LLaMa-3, and Qwen-2.5. Designed with performance and safety in mind, Peek2 is Regex-free and delivers a $ 1.11\times $ improvement in overall throughput across the entire Byte-level BPE encoding process. This algorithm runs entirely on the CPU, has stable linear complexity $ O(n) $, and provides presegmentation results identical to those of the original Regex-based pretokenizer.

Figures (4)

• Figure 1: PeekCategorize, classifying the peeked Unicode scalar value
• Figure 2: Pretokenize, the core pretokenizer algorithm
• Figure 3: Throughput performance on different tasks, more is better
• Figure 4: Time consumption on different tasks, less is better