Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding

TL;DR

This work targets hallucinations and off-target translations in MT by introducing training-free decoding objectives that contrast the true input with a contrastive source or language variant. Specifically, it defines source-contrastive decoding via $s(Y,X)=\sum_{i=1}^{|Y|}-\log\bigl(p(y_i| y_{<i}, X)-\lambda p(y_i| y_{<i}, X')\bigr)$ and language-contrastive decoding via $s(Y,X)=\sum_{i=1}^{|Y|}-\log\bigl(p(y_i| y_{<i}, X,l_y)-\sum_{l_{y'}\in L_c}\lambda p(y_i| y_{<i}, X, l_{y'})\bigr)$, with possible combination through $\lambda_{src}$ and $\lambda_{lang}$. Evaluations on M2M-100 (418M) and SMaLL-100 across 57 translation directions show chrF2 gains of about +1.1 to +1.7, plus substantial reductions in both off-target and oscillatory hallucinations (67-83% and 75-92%, respectively). The approach also transfers to LLM prompting, where language-contrastive decoding reduces off-target English outputs in prompts for Llama 2 Chat, with improvements limited by higher $\lambda_{lang}$ values. Overall, the methods provide robust, training-free improvements to translation quality and reliability, with broad applicability to multilingual MT and prompting-based translation.

Abstract

Hallucinations and off-target translation remain unsolved problems in MT, especially for low-resource languages and massively multilingual models. In this paper, we introduce two related methods to mitigate these failure cases with a modified decoding objective, without either requiring retraining or external models. In source-contrastive decoding, we search for a translation that is probable given the correct input, but improbable given a random input segment. In language-contrastive decoding, we search for a translation that is probable, but improbable given the wrong language indicator token. Experiments on the massively multilingual models M2M-100 (418M) and SMaLL-100 show that these methods suppress hallucinations and off-target translations, reducing the number of translations with segment-level chrF2 below 10 by 67-83% on average, and the number of translations with oscillatory hallucinations by 75-92% on average, across 57 tested translation directions. In a proof of concept on out-of-English translation, we also show that we can suppress off-target translations with large language models. We release our source code at https://github.com/ZurichNLP/ContraDecode.

Figures (3)

• Figure 1: Our decoding objective yields a translation that is probable given the actual input, but improbable given a source-contrastive or language-contrastive input.
• Figure 2: Off-target translation rate for Llama 2 Chat models when translating the English Flores-101 devtest set into German. Language-contrastive decoding tends to reduce off-target translation as $\lambda_{\text{lang}}$ is increased.
• Figure 3: Off-target translation rate for Llama 2 Chat models when translating the English Flores-101 devtest set into French. As with German (Figure \ref{['fig:llm-en-de']}), language-contrastive decoding tends to reduce off-target translation as $\lambda_{\text{lang}}$ is increased.