Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation

Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, Shujian Huang

TL;DR

This work analyzes how Large Language Models evaluate machine translations, focusing on the relative utility of source versus reference information. By testing four input modes (T, S-T, R-T, S-R-T) across coarse-grained scores and fine-grained error detection, the authors show that reference information consistently improves evaluation accuracy while source information can be detrimental, revealing limitations in cross-lingual transfer within current LLMs. The study includes comprehensive meta-evaluations of MQM spans and categories, a critical error-detection task, and experiments with fine-tuning using MQM data, demonstrating that while prompt-based and fine-tuning approaches can boost performance, they do not fully unleash cross-lingual capabilities. The findings highlight a clear research direction: develop methods that better exploit cross-lingual signals in LLMs for translation evaluation, with practical implications for designing more reliable, interpretable MT evaluation systems.

Abstract

This study investigates how Large Language Models (LLMs) leverage source and reference data in machine translation evaluation task, aiming to better understand the mechanisms behind their remarkable performance in this task. We design the controlled experiments across various input modes and model types, and employ both coarse-grained and fine-grained prompts to discern the utility of source versus reference information. We find that reference information significantly enhances the evaluation accuracy, while surprisingly, source information sometimes is counterproductive, indicating LLMs' inability to fully leverage the cross-lingual capability when evaluating translations. Further analysis of the fine-grained evaluation and fine-tuning experiments show similar results. These findings also suggest a potential research direction for LLMs that fully exploits the cross-lingual capability of LLMs to achieve better performance in machine translation evaluation tasks.

Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation

TL;DR

This work analyzes how Large Language Models evaluate machine translations, focusing on the relative utility of source versus reference information. By testing four input modes (T, S-T, R-T, S-R-T) across coarse-grained scores and fine-grained error detection, the authors show that reference information consistently improves evaluation accuracy while source information can be detrimental, revealing limitations in cross-lingual transfer within current LLMs. The study includes comprehensive meta-evaluations of MQM spans and categories, a critical error-detection task, and experiments with fine-tuning using MQM data, demonstrating that while prompt-based and fine-tuning approaches can boost performance, they do not fully unleash cross-lingual capabilities. The findings highlight a clear research direction: develop methods that better exploit cross-lingual signals in LLMs for translation evaluation, with practical implications for designing more reliable, interpretable MT evaluation systems.

Abstract

This study investigates how Large Language Models (LLMs) leverage source and reference data in machine translation evaluation task, aiming to better understand the mechanisms behind their remarkable performance in this task. We design the controlled experiments across various input modes and model types, and employ both coarse-grained and fine-grained prompts to discern the utility of source versus reference information. We find that reference information significantly enhances the evaluation accuracy, while surprisingly, source information sometimes is counterproductive, indicating LLMs' inability to fully leverage the cross-lingual capability when evaluating translations. Further analysis of the fine-grained evaluation and fine-tuning experiments show similar results. These findings also suggest a potential research direction for LLMs that fully exploits the cross-lingual capability of LLMs to achieve better performance in machine translation evaluation tasks.
Paper Structure (40 sections, 9 equations, 3 figures, 13 tables)

This paper contains 40 sections, 9 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automatic Translation Evaluation.
  4. LLM-based Text Evaluation.
  5. Coarse-grained Score Prediction
  6. Experimental Setup
  7. Data.
  8. Models.
  9. Evaluation Metrics.
  10. Results
  11. Fine-grained Error Detection
  12. Experimental Setup
  13. Data.
  14. Models.
  15. Meta Evaluation.
  16. ...and 25 more sections

Figures (3)

  • Figure 1: The GEMBA-SQM prompt template. The green parts will be included in the prompt if the reference information is given. Similarly, the red part will be in the prompt if the source is given. Detailed prompts can be found in Appendix \ref{['sec:detailofgemba']}.
  • Figure 2: The AutoMQM prompt template. The green parts and red parts are included according to whether the related information is given. The yellow part is also determined by the input mode. The text in the shaded area is an in-context demonstration, followed by the test sample. Detailed prompts can be found in Appendix \ref{['sec:detailofautomqm']}.
  • Figure 3: Cases of GPT-3.5's outputs. The texts in red are critical errors. Up: The model identifies the named entity error successfully. Middle: The model fails to detect the addition error. Bottom: The model fails to detect the named entity error.