Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Automatic Metrics Assess High-Quality Translations?

Sweta Agrawal, António Farinhas, Ricardo Rei, André F. T. Martins

TL;DR

This work interrogates whether automatic MT metrics can reliably identify high-quality translations beyond broad correlation with human scores, focusing on HQ translations defined by MQM judgments. By stress-testing a wide range of reference-based, quality-estimation, and LLM-based metrics on WMT MQM data, the study shows that correlations degrade when ranking translations for the same source, and many metrics fail to assign valid scores to translations with zero major errors (HQ-Zero). The GPT-4–driven GEMBA-MQM achieves the best F1 in HQ-Zero detection but exhibits biases toward GPT-4 outputs, suggesting metric biases can distort HQ identification. The findings highlight the need for robust, multi-metric evaluation protocols and caution when using automatic metrics for fine-grained ranking or system reranking in high-stakes translation tasks.

Abstract

Automatic metrics for evaluating translation quality are typically validated by measuring how well they correlate with human assessments. However, correlation methods tend to capture only the ability of metrics to differentiate between good and bad source-translation pairs, overlooking their reliability in distinguishing alternative translations for the same source. In this paper, we confirm that this is indeed the case by showing that current metrics are insensitive to nuanced differences in translation quality. This effect is most pronounced when the quality is high and the variance among alternatives is low. Given this finding, we shift towards detecting high-quality correct translations, an important problem in practical decision-making scenarios where a binary check of correctness is prioritized over a nuanced evaluation of quality. Using the MQM framework as the gold standard, we systematically stress-test the ability of current metrics to identify translations with no errors as marked by humans. Our findings reveal that current metrics often over or underestimate translation quality, indicating significant room for improvement in automatic evaluation methods.

Can Automatic Metrics Assess High-Quality Translations?

TL;DR

This work interrogates whether automatic MT metrics can reliably identify high-quality translations beyond broad correlation with human scores, focusing on HQ translations defined by MQM judgments. By stress-testing a wide range of reference-based, quality-estimation, and LLM-based metrics on WMT MQM data, the study shows that correlations degrade when ranking translations for the same source, and many metrics fail to assign valid scores to translations with zero major errors (HQ-Zero). The GPT-4–driven GEMBA-MQM achieves the best F1 in HQ-Zero detection but exhibits biases toward GPT-4 outputs, suggesting metric biases can distort HQ identification. The findings highlight the need for robust, multi-metric evaluation protocols and caution when using automatic metrics for fine-grained ranking or system reranking in high-stakes translation tasks.

Abstract

Automatic metrics for evaluating translation quality are typically validated by measuring how well they correlate with human assessments. However, correlation methods tend to capture only the ability of metrics to differentiate between good and bad source-translation pairs, overlooking their reliability in distinguishing alternative translations for the same source. In this paper, we confirm that this is indeed the case by showing that current metrics are insensitive to nuanced differences in translation quality. This effect is most pronounced when the quality is high and the variance among alternatives is low. Given this finding, we shift towards detecting high-quality correct translations, an important problem in practical decision-making scenarios where a binary check of correctness is prioritized over a nuanced evaluation of quality. Using the MQM framework as the gold standard, we systematically stress-test the ability of current metrics to identify translations with no errors as marked by humans. Our findings reveal that current metrics often over or underestimate translation quality, indicating significant room for improvement in automatic evaluation methods.
Paper Structure (15 sections, 5 figures, 7 tables)

This paper contains 15 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. How good are current MT systems?
  3. How well do MT metrics assess HQ translations?
  4. How do metrics rank HQ translations?
  5. Metrics exhibit only a low-to-fair correlation with human judgments when evaluating translations for the same source.
  6. QE metrics are on par with reference-based ones for differentiating translations.
  7. Metrics fail to distinguish HQ translations.
  8. How well do metrics detect HQ translations with no errors?
  9. Metric scores have high variance for HQ translations.
  10. Metrics over or underestimate translation quality.
  11. Which HQ translations are detected?
  12. Conclusions and Future Work
  13. Automatic Metrics
  14. Ranking results
  15. HQ-Zero Detection Results

Figures (5)

  • Figure 1: Ranking analysis configurations. $\rho$: Spearman correlation.
  • Figure 2: Top: Metric Scores distribution for HQ-Zero translations on WMT23. Bottom: Precision, recall, and F1.
  • Figure 3: Absolute difference of the number of times a metric assigns a valid score to HQ-Zero and non HQ-Zero translations.
  • Figure 4: Top: Scores distribution for HQ-Zero translations on WMT22. Bottom: Precision, recall, and F1.
  • Figure 5: Top: Scores distribution for HQ-Zero translations on WMT22 Chat. Bottom: Precision, recall, and F1.