Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier

TL;DR

The paper tackles the bias introduced by conventional pre-tokenization, where most pretokens map to single tokens (often whole words), limiting the efficacy of large vocabularies. It introduces BoundlessBPE, a single-pass extension of Byte Pair Encoding that permits supermerges of adjacent pretokens into superwords and employs IoS-based deletions to prune low-utility intermediate tokens, yielding a more uniform token distribution and improved compression (bytes per token up to 9–15% higher). The authors also present BOUNDLESS_PATTERN, a pre-tokenization regex designed to work with superwords, enabling better handling of code identifiers and multi-word expressions, particularly for named entities. Together, these contributions aim to enhance tokenizer efficacy, potentially improving downstream model performance by enabling more effective use of larger vocabularies and more balanced token frequencies.

Abstract

Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. While this process encourages having full, individual words as tokens, it introduces a fundamental limitation in most tokenization algorithms such as Byte Pair Encoding (BPE). Specifically, pre-tokenization causes the distribution of tokens in a corpus to heavily skew towards common, full-length words. This skewed distribution limits the benefits of expanding to larger vocabularies, since the additional tokens appear with progressively lower counts. To overcome this barrier, we propose BoundlessBPE, a modified BPE algorithm that relaxes the pretoken boundary constraint. Our approach selectively merges two complete pretokens into a larger unit we term a superword. Superwords are not necessarily semantically cohesive. For example, the pretokens " of" and " the" might be combined to form the superword " of the". This merging strategy results in a substantially more uniform distribution of tokens across a corpus than standard BPE, and compresses text more effectively, with up to a 15% increase in bytes per token.

Figures (13)

• Figure 1: Proportion of pretokens in the evaluation corpus that are represented as full tokens by different tokenization methods (BPE, UnigramLM, WordPiece) across varying vocabulary sizes.
• Figure 2: Logarithm of the maximum count of each selected merge, up to a vocabulary size of 131,072.
• Figure 3: Average fraction of merges (lower) and supermerges (upper) over intervals of 8,192 merges, up to a vocabulary size of 131,072.
• Figure 4: The left column is the $\log_{10}(\text{count}+1)$ for each token, sorted from most to least frequent on the $x$-axis on our evaluation corpus. The +1 is to allow plotting of 0 counts. The right column shows a zoomed-in view of the 250 most common tokens.
• Figure 5: Fraction of vocabulary used at least once in an evaluation corpus, across different tokenization methods and vocabulary sizes. A higher fraction suggests the vocabulary has more useful tokens for representing unseen data.