Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies

TL;DR

This paper tackles the lack of a single dominant MT evaluation metric by quantifying how metric deltas translate into human judgments. Using the ToShip23 dataset and complementary WMT data, it derives a delta-accuracy framework that maps differences between systems across multiple metrics into a unified accuracy space via a sigmoid fit, revealing that some metrics (notably CometKiwi_22^QE) align with human judgments at small deltas, while others (like BLEU) require large deltas and still struggle to reach high agreement, especially for unrelated systems. The authors provide practical thresholds, validate them across domains and language pairs, and offer a tool for cross-metric interpretation, advocating reporting metric-delta accuracy alongside traditional significance testing. Collectively, the work helps MT researchers and practitioners interpret improvements more reliably, guides metric selection (favoring QE-based metrics), and highlights the limitations of string-based metrics for diverse or unrelated system comparisons. The approach emphasizes stability of delta-accuracy over testset size and grounds automatic evaluation in human perceptual judgments, with broad implications for standardizing MT evaluation practice.

Abstract

Ten years ago a single metric, BLEU, governed progress in machine translation research. For better or worse, there is no such consensus today, and consequently it is difficult for researchers to develop and retain the kinds of heuristic intuitions about metric deltas that drove earlier research and deployment decisions. This paper investigates the "dynamic range" of a number of modern metrics in an effort to provide a collective understanding of the meaning of differences in scores both within and among metrics; in other words, we ask what point difference X in metric Y is required between two systems for humans to notice? We conduct our evaluation on a new large dataset, ToShip23, using it to discover deltas at which metrics achieve system-level differences that are meaningful to humans, which we measure by pairwise system accuracy. We additionally show that this method of establishing delta-accuracy is more stable than the standard use of statistical p-values in regards to testset size. Where data size permits, we also explore the effect of metric deltas and accuracy across finer-grained features such as translation direction, domain, and system closeness.

Figures (13)

• Figure 1: Distribution of pairwise system deltas for each metric over all systems from WMT22. Gray rectangles show min-max range which is vastly different between metrics. Standard deviations (black lines) also differ.
• Figure 2: What pairwise accuracy (left-y-axis) to expect when seeing given certain acceptance threshold (x-axis). The bin width (right-y-axis) shows the width of the bin for metric delta that contains 300 system pairs.
• Figure 3: Empirical pairwise accuracies for various metrics with a fitted sigmoid curves on ToShip23 dataset. All metrics are in \ref{['fig:expected-accuracy-with_sigmoid_big']}.
• Figure 4: Testing the validity of thresholds devised on ToShip23 with WMT datasets. In a scenario without noisy data, we would expect the real accuracies to match the estimated accuracies (the black line). See detailed per-metric breakdown in \ref{['fig:test_on_wmt_big']}.
• Figure 5: Comparison of pairwise accuracy on ToShip23 dataset when comparing into English, out-of-English, and Chinese, Japanese, Korean language pairs separately. The count shows total number of system-pairs in the evaluation. See other metrics in \ref{['fig:XE_EX_comparison_big']}.