TASER: Translation Assessment via Systematic Evaluation and Reasoning

TL;DR

The paper tackles the challenge of translating quality assessment by leveraging Large Reasoning Models (LRMs) to perform explicit, step-by-step evaluations. TASER combines structured prompts with LRMs to assess translation quality in both reference-based and reference-free settings, achieving state-of-the-art system-level performance on the WMT24 MQM benchmark and competitive segment-level results. A key finding is that structured prompting and the depth of explicit reasoning influence evaluation outcomes, with low-reasoning variants performing exceptionally well in the reference-free mode. The work highlights the interpretability benefits of explicit reasoning in MT evaluation and suggests that reasoning-based metrics can enhance reliability and trust in automated translation quality assessment, while noting limitations related to closed-source models and data concerns.

Abstract

We introduce TASER (Translation Assessment via Systematic Evaluation and Reasoning), a metric that uses Large Reasoning Models (LRMs) for automated translation quality assessment. TASER harnesses the explicit reasoning capabilities of LRMs to conduct systematic, step-by-step evaluation of translation quality. We evaluate TASER on the WMT24 Metrics Shared Task across both reference-based and reference-free scenarios, demonstrating state-of-the-art performance. In system-level evaluation, TASER achieves the highest soft pairwise accuracy in both reference-based and reference-free settings, outperforming all existing metrics. At the segment level, TASER maintains competitive performance with our reference-free variant ranking as the top-performing metric among all reference-free approaches. Our experiments reveal that structured prompting templates yield superior results with LRMs compared to the open-ended approaches that proved optimal for traditional LLMs. We evaluate o3, a large reasoning model from OpenAI, with varying reasoning efforts, providing insights into the relationship between reasoning depth and evaluation quality. The explicit reasoning process in LRMs offers interpretability and visibility, addressing a key limitation of existing automated metrics. Our results demonstrate that Large Reasoning Models show a measurable advancement in translation quality assessment, combining improved accuracy with transparent evaluation across diverse language pairs.