Chain-of-Thought Reasoning Improves Context-Aware Translation with Large Language Models

TL;DR

The paper investigates whether chain-of-thought prompting improves context-aware translation by testing 12 large language models on DiscEvalMT across two tasks: contrastive selection of correct translations and generation-based translation. It compares reasoning-enabled prompts against non-reasoning prompts, finding that CoT reasoning substantially boosts performance for the largest models (e.g., GPT-4o, GPT-4, Phi-4), achieving about 90% ACC on the contrastive task and COMET scores around 0.92 on translation, with a consistent 'wise get wiser' pattern across models. The work demonstrates state-of-the-art results on English–French DiscEvalMT for pronominal anaphora and lexical cohesion and discusses practical implications for deploying reasoning in MT pipelines, including potential document-level and agentic AI applications. Overall, the findings advocate targeted use of CoT reasoning to improve coherence-dependent translation, particularly for capable LLMs, and point to future directions in identifying where reasoning provides the most benefit and how to structure it for efficient parsing and integration.

Abstract

This paper assesses the capacity of large language models (LLMs) to translate texts that include inter-sentential dependencies. We use the English-French DiscEvalMT benchmark (Bawden et al., 2018) with pairs of sentences containing translation challenges either for pronominal anaphora or for lexical cohesion. We evaluate 12 LLMs from the DeepSeek-R1, GPT, Llama, Mistral and Phi families on two tasks: (1) distinguishing a correct translation from a wrong but plausible one; (2) generating a correct translation. We compare prompts that encourage chain-of-thought reasoning with those that do not. The best models take advantage of reasoning and reach about 90% accuracy on the first task, and COMET scores of about 92% on the second task, with GPT-4, GPT-4o and Phi standing out. Moreover, we observe a "wise get wiser" effect: the improvements through reasoning are positively correlated with the scores of the models without reasoning.

Figures (20)

• Figure 1: Mean accuracy (top bars) and inconsistency (bottom bars, smaller are better) on the contrastive anaphora task. For each LLM, the four bars are those of the prompts: (none, simple), (simple, simple), (detailed, simple) and (simple, step-by-step). LLMs are ranked by decreasing scores with the last prompt (yellow), which reaches highest overall performance.
• Figure 1: Mean accuracy (top) and inconsistency (bottom; lower is better) across prompt configurations for each model on the anaphora validation set. Reproduced here from the main text to keep the appendix self-contained.
• Figure 2: Correlations between $\Delta$ values (reasoning minus no_reasoning) obtained with various MT metrics for 12 LLMs, on anaphora translation. Each cell shows Pearson (upper part) and Spearman (lower part) correlations.
• Figure 2: Average response time per prompt for open-source models on anaphora. Larger models tend to be slower; complex prompts increase latency.
• Figure 3: Average response time per prompt for OpenAI models on anaphora. GPT-4 is slowest overall, especially with the reasoning prompt.