Paying More Attention to Source Context: Mitigating Unfaithful Translations from Large Language Model

TL;DR

This paper tackles unfaithful translations produced by decoder-only LLMs, which lack explicit source-target alignment. It introduces three complementary strategies: reweighting source-context attention to strengthen source influence, contrastive decoding to downweight target-prefix bias, and target-constrained instruction tuning to reduce reliance on target prefixes. Across zeroshot and supervised settings, the methods yield consistent gains in BLEU and COMET on unfaithful translation test sets and general benchmarks, with human evaluation showing substantial reductions in hallucinations. The findings highlight that guiding LLMs to attend more to source content improves translation faithfulness and suggests that these techniques can generalize to other seq2seq tasks and model families.

Abstract

Large language models (LLMs) have showcased impressive multilingual machine translation ability. However, unlike encoder-decoder style models, decoder-only LLMs lack an explicit alignment between source and target contexts. Analyzing contribution scores during generation processes revealed that LLMs can be biased towards previously generated tokens over corresponding source tokens, leading to unfaithful translations. To address this issue, we propose to encourage LLMs to pay more attention to the source context from both source and target perspectives in zeroshot prompting: 1) adjust source context attention weights; 2) suppress irrelevant target prefix influence; Additionally, we propose 3) avoiding over-reliance on the target prefix in instruction tuning. Experimental results from both human-collected unfaithfulness test sets focusing on LLM-generated unfaithful translations and general test sets, verify our methods' effectiveness across multiple language pairs. Further human evaluation shows our method's efficacy in reducing hallucinatory translations and facilitating faithful translation generation.

Figures (9)

• Figure 1: Contribution visualization of a Zh$\Rightarrow$En unfaithful translation instance. Each predicted token (row) corresponds to the contribution of each input token including source tokens and target prefixes (column) to the output token. One of the correct translations of the given source sentence is "We just want you to honestly and sincerely say Sorry".
• Figure 2: The left picture shows the paradigms of proposed unsupervised methods, including the reweight attention and contrastive decoding. The right picture illustrates the target-constrained tuning, detailing how the two different inputs, full input $x$ and label-masked input $x_{m}$ will go through the model and obtain two distributions $P_1$ and $P_2$.
• Figure 3: Human annotation results: percentages of translation categories for different methods.
• Figure 4: For each generation step, the figure shows the (a) contribution and (b) entropy of source context in the translation direction from Chinese to English. The points on the lines denotes the average score across the $i$-th target token position. Note that different methods result in different target generation lengths.
• Figure 5: Contribution visualization for unfaithful translations and corresponding mitigation across various settings. One of the correct translations of the first row is: "Another poet friend of mine has a slogan."; One of the correct translations of the second row is: "Just do this to your TV screen, rub it from the center to the four corners."