Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost

TL;DR

The paper systematically evaluates large reasoning models as MT quality evaluators within the MQM framework, uncovering material misalignment, post-scoring pitfalls, and inefficient thinking budgets. It introduces ThinMQM, a lightweight calibration approach that trains LRMs on synthetic human-like MQM trajectories to align their reasoning with human evaluation processes. Across WMT24 metrics, ThinMQM substantially reduces thinking budgets (approximately 35x) while improving evaluation performance, with notable gains such as +8.7 correlation points for 7B models and competitive performance against state-of-the-art metrics like xCOMET. The work demonstrates that with careful calibration and thinking-trajectory training, LRMs hold strong potential as efficient, fine-grained MT evaluators, while highlighting the need for controlled reasoning and robust evaluation material design for scalable deployment.

Abstract

Recent advancements in large reasoning models (LRMs) have introduced an intermediate "thinking" process prior to generating final answers, improving their reasoning capabilities on complex downstream tasks. However, the potential of LRMs as evaluators for machine translation (MT) quality remains underexplored. We provides the first systematic analysis of LRM-as-a-judge in MT evaluation. We identify key challenges, revealing LRMs require tailored evaluation materials, tend to "overthink" simpler instances and have issues with scoring mechanisms leading to overestimation. To address these, we propose to calibrate LRM thinking by training them on synthetic, human-like thinking trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this approach largely reduces thinking budgets by ~35x while concurrently improving evaluation performance across different LRM scales from 7B to 32B (e.g., R1-Distill-Qwen-7B achieves a +8.7 correlation point improvement). These findings highlight the potential of efficiently calibrated LRMs to advance fine-grained automatic MT evaluation.

Figures (14)

• Figure 1: Performance comparison of various evaluation models on WMT24 metrics tasks. The ThinMQM models (Ours) achieve strong performance with competitive efficiency. "S, R, J" denote different evaluation inputs: Source-only, Reference-based, and Joint evaluation materials, respectively.
• Figure 2: Overview of our research framework. (a) We decompose the general LRMs-as-judge pipeline for MT evaluation and identify key issues. (b) Guided by analytical findings, we propose ThinMQM, which establishes a more effective reasoning process.
• Figure 2: Effect of changing the rule-based scoring weights on average correlation Avg. $\Delta$ ($Acc^*_{eq}$., SPA) metrics.
• Figure 3: Shapley value analysis of input contributions of evaluation materials at different evaluation granularities. "R1D-L./Q." refers to the R1-Distill-Llama/-Qwen models.
• Figure 4: Multi-run evaluation of ThinMQM at temperature 0.6. Each score is presented as ${\mathrm{Mean}}_{\mathrm{Std.}}$ over 3 independent runs.