LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation

TL;DR

This work analyzes why zero-shot multilingual NMT often translates into the wrong target language due to the placement of language tags. It shows encoder-side language indication can be unstable and that top encoder layers tend to encode target-language signals most effectively. The authors propose Language Converter Strategy (LCS), which splits the encoder into shallow and deep language-converter layers and injects target-language embeddings into the converter layers, improving language accuracy and zero-shot BLEU without adding parameters. Across MultiUN, TED, and OPUS-100 (including noisy data), LCS achieves substantial zero-shot gains and strengthens the signaling of the target language, while remaining compatible with existing methods. The approach offers a practical, scalable improvement for reliable zero-shot translation in MNMT systems.

Abstract

Multilingual neural machine translation models generally distinguish translation directions by the language tag (LT) in front of the source or target sentences. However, current LT strategies cannot indicate the desired target language as expected on zero-shot translation, i.e., the off-target issue. Our analysis reveals that the indication of the target language is sensitive to the placement of the target LT. For example, when placing the target LT on the decoder side, the indication would rapidly degrade along with decoding steps, while placing the target LT on the encoder side would lead to copying or paraphrasing the source input. To address the above issues, we propose a simple yet effective strategy named Language Converter Strategy (LCS). By introducing the target language embedding into the top encoder layers, LCS mitigates confusion in the encoder and ensures stable language indication for the decoder. Experimental results on MultiUN, TED, and OPUS-100 datasets demonstrate that LCS could significantly mitigate the off-target issue, with language accuracy up to 95.28%, 96.21%, and 85.35% meanwhile outperforming the vanilla LT strategy by 3.07, 3,3, and 7.93 BLEU scores on zero-shot translation, respectively.

Figures (7)

• Figure 1: Fine-grained language rate of the desired target language (i.e., language accuracy) and undesired English throughout decoding steps on the zero-shot testset of denoised OPUS-100, with 5 words in each interval of the final translation results.
• Figure 2: Curves of the similarity of the language pairs along encoder layers on the zero-shot testset of denoised OPUS-100. The higher similarity denotes the representation is more similar and language-agnostic.
• Figure 3: Illustration of the encoder of LCS. For the target, only the language tag could be seen by the encoder.
• Figure 4: Curves of the similarity of the language pairs along encoder layers in S-Enc-T-Dec on the zero-shot testset of noise and denoised OPUS-100.
• Figure 5: Performance of deeper encoder on the supervised and zero-shot testset of the noise OPUS-100.