Theoretical Analysis of Byte-Pair Encoding

TL;DR

This paper focuses on the optimization problem underlying BPE: finding a pair encoding that achieves optimal compression utility, and shows that this problem is APX-complete, indicating that it is unlikely to admit a polynomial-time approximation scheme.

Abstract

Byte-Pair Encoding (BPE) is a widely used method for subword tokenization, with origins in grammar-based text compression. It is employed in a variety of language processing tasks such as machine translation or large language model (LLM) pretraining, to create a token dictionary of a prescribed size. Most evaluations of BPE to date are empirical, and the reasons for its good practical performance are not well understood. In this paper we focus on the optimization problem underlying BPE: finding a pair encoding that achieves optimal compression utility. We show that this problem is APX-complete, indicating that it is unlikely to admit a polynomial-time approximation scheme. This answers, in a stronger form, a question recently raised by Zouhar et al. On the positive side, we show that BPE approximates the compression utility of the optimal pair encoding to a worst-case factor between $0.333$ and $0.625$. Our results aim to explain the ongoing success of BPE and are, to our knowledge, the first rigorous guarantees on its compression utility that hold for all inputs.

Figures (3)

• Figure 1: Input $s = \mathtt{aabaaaba}$ encoded by BPE merge sequence $(\mathtt{aa \rightarrow X,\, Xb \rightarrow Y,\, YX \rightarrow Z)}$ (above). An optimal encoding by the merge sequence $(\mathtt{ab \rightarrow X,\, aX \rightarrow Y,\, Ya \rightarrow Z})$ (below).
• Figure 2: Input $s = \mathtt{abcd\,|\,bc\,|\,bcda\,|\,cd\,|\,cdab\,|\,da\,|\,dabc\,|\,ab}$, where $|$ denotes a distinct symbol for each occurrence. An optimal OPE encoding of $s$ (above) with utility $\mathop{\mathrm{\mathsf{OPT}}}\nolimits(s,4) = 12$. An optimal OMS encoding of $s$ (below) with utility $\mathop{\mathrm{\mathsf{OPT}}}\nolimits^{m}(s,4)=11$. The OMS solution is obtained via the merge sequence $(\mathtt{ab \rightarrow X,~ bc \rightarrow Y,~ cd \rightarrow Z,~ da \rightarrow T})$.
• Figure 3: Encoding $s = \mathtt{abcd\,|\,bc\,|\,bcda\,|\,cd\,|\,cdab\,|\,da\,|\,dabc\,|\,ab}$, where $|$ denotes a distinct symbol for each occurrence. The BPE encoding (above) with utility $\mathop{\mathrm{\mathsf{BPE}}}\nolimits(s,4) = 10$ via the merge sequence $(\mathtt{ab \rightarrow X,~ cd \rightarrow Y,~ bc \rightarrow Z,~ da \rightarrow T})$. Arrows show the $k$-packing solution derived from the BPE merge sequence. The EvenOdd encoding (below) with $k=4$ and utility $12$. Arrows show the indices of most frequent pairs, with crossed ones removed.

Theorems & Definitions (12)

• Theorem 1.1
• Theorem 1.2
• Theorem 1.3
• Theorem 1.4
• Claim 2.1
• proof
• Lemma 3.1
• proof
• Lemma 3.2
• proof