Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation

TL;DR

The paper addresses the challenge of evaluating machine translation when non-literal language is prevalent, introducing the MENT meta-evaluation dataset to systematically assess MT metrics in SNS, cross-cultural, poetry, and literature domains. It demonstrates that traditional metrics and static LLM-based judges struggle with non-literal content due to semantic nuance and knowledge cutoffs, motivating the RATE framework. RATE employs a Core Agent to dynamically orchestrate sub-agents—Evaluation, Search, and Comparison—to retrieve background knowledge and calibrate judgments, achieving at least a 3.2 meta-score improvement on MENT and showing robustness on general-domain evaluation. The work advances translation evaluation by enabling adaptive, knowledge-grounded, and calibrated assessments, and provides code and data to foster further research in non-literal MT evaluation.

Abstract

Large Language Models (LLMs) have significantly advanced Machine Translation (MT), applying them to linguistically complex domains-such as Social Network Services, literature etc. In these scenarios, translations often require handling non-literal expressions, leading to the inaccuracy of MT metrics. To systematically investigate the reliability of MT metrics, we first curate a meta-evaluation dataset focused on non-literal translations, namely MENT. MENT encompasses four non-literal translation domains and features source sentences paired with translations from diverse MT systems, with 7,530 human-annotated scores on translation quality. Experimental results reveal the inaccuracies of traditional MT metrics and the limitations of LLM-as-a-Judge, particularly the knowledge cutoff and score inconsistency problem. To mitigate these limitations, we propose RATE, a novel agentic translation evaluation framework, centered by a reflective Core Agent that dynamically invokes specialized sub-agents. Experimental results indicate the efficacy of RATE, achieving an improvement of at least 3.2 meta score compared with current metrics. Further experiments demonstrate the robustness of RATE to general-domain MT evaluation. Code and dataset are available at: https://github.com/BITHLP/RATE.

Figures (24)

• Figure 1: An illustration of evaluation misalignment in traditional metrics whereas the alignment with RATE. Traditional metrics often over-score literal but semantically incorrect translation (left) and penalize idiomatic translation (right). In contrast, our RATE leverages sub-agents to retrieve background knowledge and score calibration, achieving alignment with human judgment.
• Figure 2: Overview of the data construction pipeline and final dataset visualization. (Left) The four-stage construction pipeline, proceeding from multi-domain sentences collection to strict quality control. (Right) Visualization of the MENT dataset, illustrating the distribution of source domains and linguistic phenomena.
• Figure 3: Heatmap of Pearson correlations for system-level inter-annotator agreement. Each cell $(i, j)$ displays the correlation coefficient calculated on the aggregated scores of shared translations. Blank cells denote pairs with no overlapping annotation tasks due to the workload distribution.
• Figure 4: Overview of the RATE framework. The Core Agent acts as the central controller, dynamically selecting specialized sub-agents (Evaluation, Search, and Comparison) based on current state and outputs of sub-agents to iteratively refine the translation evaluation.
• Figure 5: Illustration of metrics performance with specific domain, detailed numerical results in Appendix \ref{['sec:appendix_domain_eval_result']}.