An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

TL;DR

The paper investigates the common practice of trimming rare subwords from Byte-Pair Encoding vocabularies in neural machine translation. Using a controlled Transformer setup on the IWSLT14 German-English task (and a larger EN-FR dataset), it evaluates ten hypotheses across split and joint vocabularies, multiple base sizes, and various thresholds, always comparing trimmed vs baseline models. Across extensive experiments, trimming rarely improves BLEU and often degrades performance, with only isolated marginal gains under specific configurations; in many cases, trimming reduces effective vocabulary size and increases sequence length without commensurate gains. The findings suggest that, in transformer-based MT, the practical benefits of vocabulary trimming (parameter reduction, robustness) do not offset the adverse effects on modeling quality, and the practice should be deprioritized in favor of maintaining robust, larger vocabularies when feasible. The work also contextualizes trimming historically, arguing that benefits prominent in older architectures do not translate to modern large-scale models.

Abstract

We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in machine translation implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to improve performance, and is even prone to incurring heavy degradation.

Figures (2)

• Figure 1: An example BPE tokenization. The shaded region contains intermediate subwords, which often appear during the merging process to build longer subwords, but rarely in the final tokenization of a word.
• Figure 2: An example of a trimmed tokenizer during inference. The left side shows the original character sequence and the final BPE-tokenized sequence, token·ization. On the right, tokens are decomposed if they have less than the designated threshold, according to the function dec. The tokens ization = (iz, ation) and iz = (i, z) are in $\mathcal{X}_{\mathcal{B}, T}$ and are decomposed, resulting in token·i·z·ation.