Neural Machine Translation with Byte-Level Subwords

TL;DR

The paper introduces BBPE, a byte-level subword vocabulary learned over UTF-8 byte n-grams to compress vocab size while maintaining translation quality. It demonstrates that contextualizing BBPE embeddings with either depthwise convolution or BiGRU yields substantial gains, especially for finer-grained vocabularies, and achieves competitive BLEU scores with significantly smaller vocabularies than BPE. The approach enables cross-language sharing and transfer learning between languages with non-overlapping character sets, highlighting practical benefits for multilingual MT and model reuse. Overall, BBPE offers a compact, transferable alternative to character and word-level vocabularies with favorable efficiency and applicability to diverse language settings.

Abstract

Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords or words. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes and using the 256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is. We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE. In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.

Figures (5)

• Figure 1: BPE (upper) and BBPE (lower) tokenization of a Japanese sentence. Bytes (from partial characters) are represented by hexadecimal digits.
• Figure 2: Symbol frequencies (in log2 scale) for En-De (left) and X-En (right) vocabularies. BBPE enables a more consistent distribution of vocabulary across frequencies.
• Figure 3: The numbers of languages symbols have shared across Ar, He, Ru, Ko and It (from X-En). Note that these languages have mutually different character sets.
• Figure 4: X-En validation BLEU for models without contextualization, with local contextualization (depth-wise convolution) and with long-range contextualization (Bi-GRU). The y axis starts from 28.2 to focus on the gain portions and facilitate comparison across different vocabularies.
• Figure 5: An example from Ja-En tokenized with different vocabularies. Raw spaces are replaced by underscores and spaces are used to split tokens. We can observe how tokens look like as the tokenization granularity goes from fine to coarse: Byte (256) $\rightarrow$ BBPE (1K, 2K, 4K, 8K) $\rightarrow$ Char (8K) $\rightarrow$ BBPE (16K, 32K) $\rightarrow$ BPE (16K, 32K).