Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

Stefano Perrella, Lorenzo Proietti, Pere-Lluís Huguet Cabot, Edoardo Barba, Roberto Navigli

TL;DR

This work introduces an interpretable evaluation framework for Machine Translation evaluation metrics, and measures the performance of MT metrics using Precision, Recall, and F-score to offer clearer insights into their capabilities than correlation with human judgments.

Abstract

Machine Translation (MT) evaluation metrics assess translation quality automatically. Recently, researchers have employed MT metrics for various new use cases, such as data filtering and translation re-ranking. However, most MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. Moreover, MT metrics' capabilities have historically been evaluated using correlation with human judgment, which, despite its efficacy, falls short of providing intuitive insights into metric performance, especially in terms of new metric use cases. To address these issues, we introduce an interpretable evaluation framework for MT metrics. Within this framework, we evaluate metrics in two scenarios that serve as proxies for the data filtering and translation re-ranking use cases. Furthermore, by measuring the performance of MT metrics using Precision, Recall, and F-score, we offer clearer insights into their capabilities than correlation with human judgments. Finally, we raise concerns regarding the reliability of manually curated data following the Direct Assessments+Scalar Quality Metrics (DA+SQM) guidelines, reporting a notably low agreement with Multidimensional Quality Metrics (MQM) annotations.

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics

TL;DR

This work introduces an interpretable evaluation framework for Machine Translation evaluation metrics, and measures the performance of MT metrics using Precision, Recall, and F-score to offer clearer insights into their capabilities than correlation with human judgments.

Abstract

Machine Translation (MT) evaluation metrics assess translation quality automatically. Recently, researchers have employed MT metrics for various new use cases, such as data filtering and translation re-ranking. However, most MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. Moreover, MT metrics' capabilities have historically been evaluated using correlation with human judgment, which, despite its efficacy, falls short of providing intuitive insights into metric performance, especially in terms of new metric use cases. To address these issues, we introduce an interpretable evaluation framework for MT metrics. Within this framework, we evaluate metrics in two scenarios that serve as proxies for the data filtering and translation re-ranking use cases. Furthermore, by measuring the performance of MT metrics using Precision, Recall, and F-score, we offer clearer insights into their capabilities than correlation with human judgments. Finally, we raise concerns regarding the reliability of manually curated data following the Direct Assessments+Scalar Quality Metrics (DA+SQM) guidelines, reporting a notably low agreement with Multidimensional Quality Metrics (MQM) annotations.
Paper Structure (34 sections, 4 equations, 5 figures, 12 tables)

This paper contains 34 sections, 4 equations, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. The Interpretability of MT Metrics' Assessments
  3. An Interpretable Evaluation Framework for MT Metrics
  4. Metrics as Binary Classifiers for Data Filtering
  5. Precision, Recall, and $\boldsymbol{F}$-score
  6. Metrics as Utility Functions for Translation Re-Ranking
  7. Experimental Setup
  8. The Data
  9. The Metrics
  10. Selecting the Thresholds tau
  11. Extracting Binary Labels from Manually-Annotated Datasets
  12. Results
  13. Binary Classification
  14. Thresholds reliability
  15. What is the human performance?
  16. ...and 19 more sections

Figures (5)

  • Figure 1: Quality assessments returned by cometrei-etal-2020-comet, MetricX-23-QE-XL juraska-etal-2023-metricx, and gemba-mqmkocmi-federmann-2023-gemba for the provided machine-translated text.
  • Figure 2: Distribution of the MQM score $\Delta$ between the openly available metrics' false positive MQM scores and human thresholds, i.e., $-4$ for Good and $-1$ for Perfect. The dataset employed is the zh$\rightarrow$en split of WMT23$_{\text{MQM}}$. Additional metrics are included in Figure \ref{['fig:fps-deltas-apx']} in the Appendix.
  • Figure 3: Tested metrics' optimal threshold values across different language directions. The thresholds were selected to maximize the $F$-score on the test set in the Good vs Bad binary classification scenario. Thresholds are normalized between $0$ and $1$ for improved clarity.
  • Figure 4: Tested metrics' optimal threshold values across different language directions. The thresholds were selected to maximize the $F$-score on the test set in the Perfect vs Other binary classification scenario. Thresholds are normalized between $0$ and $1$ for improved clarity.
  • Figure 5: Distribution of the MQM score $\Delta$ between metrics' false positive MQM scores and human thresholds, i.e., $-4$ for Good and $-1$ for Perfect. The dataset is the zh$\rightarrow$en split of WMT23$_{\text{MQM}}$.