Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation

TL;DR

This work tackles the persistent off-target issue in multilingual neural machine translation by introducing Registering, a decoder-only approach that inserts registers to map source token semantics into the target language space. By carefully redesigning the attention mask so that generation relies on registers rather than directly on the source, the method enforces generation within the desired language space and dramatically reduces off-target translations. The authors demonstrate state-of-the-art results on the EC-40 benchmark and scale up with two pre-trained MITRE models (466M and 913M parameters) trained on 9.3B sentence pairs across 24 languages, with MITRE-913M outperforming NLLB-3.3B and competing with GPT-4o mini in several directions. They also show strong fine-tuning adaptability and provide open-source releases to foster further MNMT research, highlighting the practical potential of cost-efficient MNMT that approaches LLM-like performance.

Abstract

The multilingual neural machine translation (MNMT) aims for arbitrary translations across multiple languages. Although MNMT-specific models trained on parallel data offer low costs in training and deployment, their performance consistently lags behind that of large language models (LLMs). In this work, we introduce registering, a novel method that enables a small MNMT-specific model to compete with LLMs. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method advances the state-of-the-art of MNMT. We further pre-train two models, namely MITRE (multilingual translation with registers), by 9.3 billion sentence pairs across 24 languages collected from public corpora. One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: https://github.com/zhiqu22/mitre.

Figures (10)

• Figure 1: Illustration of the attention view among different architectures. "Token" refers to the representation corresponding to the token.
• Figure 2: Illustration of registering. The example depicts a translation from English to German. The illustrated model stacks N layers, each following the Transformer decoder layer structure transformer with pre-normalization pre_norm. Notably, each circle represents a token and its representation in the generation.
• Figure 3: The spBLEU score variations on EC-40 where the x-axis is $\text{len}(\boldsymbol{x}')/\text{len}(\boldsymbol{r})$, where only the length of $\boldsymbol{r}$ is changed and $\boldsymbol{x}'$ is fixed.
• Figure 4: 2D distribution of token-level representations extracted from the output of the 24th layer of a model trained on EC-40. Each class listed in the legend contains 300 randomly sampled tokens. Appendix \ref{['appendix:rep']} shows the representational distributions from other layers.
• Figure 5: Token-level attention weights illustration, where the weight of each token is averaged across all heads of a model trained on EC-40. \ref{['fig:inseparable']} and \ref{['fig:separable']} illustrate two instances translated from de to en. The top-3 attention directions for each token are labeled, with darker colors indicating higher attention weights. Note that while the target tokens for these two instances are identical, their source tokens are not, because the verbs in \ref{['fig:inseparable']} and \ref{['fig:separable']} are semantically equivalent but have different forms. To aid understanding, we highlight the verbs with purple borders: "öffne" in \ref{['fig:inseparable']} and "mache ... auf" in \ref{['fig:separable']} both correspond to the target verb "open". Then, the registers with the highest attention weights associated with these verbs are also marked with purple borders.