Evaluating Automatic Metrics with Incremental Machine Translation Systems

TL;DR

This paper presents a six-year, weekly dataset of commercial MT outputs across 12 language directions to evaluate automatic MT metrics by their preference for newer translations, leveraging the common assumption that commercial systems improve over time. It compares surface-level, embedding-based, and trained-with-human-judgments metrics, finding that neural metrics like COMET-22 and UniTE exhibit higher time-based consistency and ranking accuracy than non-neural ones. The study also probes how metric reliability shifts with MT quality and demonstrates that synthetic references can, in some cases, match human references for metric evaluation. Overall, the dataset provides a scalable, real-world testbed for MT metric evaluation and reinforces evolving views on metric reliability in relation to MT quality.

Abstract

We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. Since human A/B testing is commonly used, we assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations. Our study not only confirms several prior findings, such as the advantage of neural metrics over non-neural ones, but also explores the debated issue of how MT quality affects metric reliability--an investigation that smaller datasets in previous research could not sufficiently explore. Overall, our research demonstrates the dataset's value as a testbed for metric evaluation. We release our code at https://github.com/gjwubyron/Evo

Figures (8)

• Figure 1: The Spearman correlation measures the relationship between metric score rankings and time rankings for MT systems. A positive correlation indicates an upward trend, with a higher correlation indicating a stronger trend. A red star indicates lack of statistical significance (p-value $>$ 0.05).
• Figure 2: Accuracy over a rolling window of 36 systems. The x-axis represents the index of the starting system, with systems ordered chronologically from the earliest to the most recent. The x-axis scale may vary due to differing numbers of systems, as discussed in Section \ref{['sec:data']}.
• Figure 3: Accuracy across three language pairs using either human or synthetic references. The two reference-free metrics are not included as they will not be influenced by reference.
• Figure 4: The metric scores for English$\rightarrow$Spanish translations. While the earliest system achieved nearly perfect scores, subsequent systems showed a notable decline.
• Figure 5: Metric scores over time.