Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration

TL;DR

This paper addresses the problem that Kendall's tau mis-handles ties in machine translation metric meta-evaluation, potentially biasing metric rankings as outputs become more similar and scores discrete. It proposes a tie-aware framework based on pairwise accuracy with ties and a tie calibration procedure to enable fair comparison across metrics. The authors validate the approach on WMT'22 MQM data across en-de, zh-en, and en-ru using several metrics (e.g., Metric-X, COMET-22, BLEURT-20, MaTESe, GEMBA) and show that the calibrated accuracy-based ranking (and its related variant) yields fairer, more robust metric rankings than traditional tau variants, and that tying can be exploited by some methods unless accounted for. They also analyze generalization, tie introduction patterns, and class-specific statistics, arguing for broader applicability to metric meta-evaluation beyond MT.

Abstract

Kendall's tau is frequently used to meta-evaluate how well machine translation (MT) evaluation metrics score individual translations. Its focus on pairwise score comparisons is intuitive but raises the question of how ties should be handled, a gray area that has motivated different variants in the literature. We demonstrate that, in settings like modern MT meta-evaluation, existing variants have weaknesses arising from their handling of ties, and in some situations can even be gamed. We propose instead to meta-evaluate metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties, in combination with a tie calibration procedure that automatically introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. We argue and provide experimental evidence that these modifications lead to fairer ranking-based assessments of metric performance.

Figures (7)

• Figure 1: Pearson's $r$, Spearman's $\rho$, and Kendall's $\tau_b$ calculated between hypothetical human scores and metric scores. Lines between data points are shown for visualization purposes.
• Figure 2: When considering ties, $m_1$ only incorrectly ranks 1 out of the $\binom{6}{2}$ pairs, whereas $m_2$ incorrectly ranks 6. However, due to how each $\tau$ handles ties, only $\textrm{acc}_\textrm{eq}$ and $\tau_\textrm{eq}$ strongly prefer $m_1$ over $m_2$. Notably, $\tau_{10}$, $\tau_{13}$, and $\tau_{14}$ are unable to distinguish a perfect metric $(m=h)$ from $m_2$. The $\textrm{acc}_\textrm{eq}$ and $\tau_\textrm{eq}$ statistics are proposed in this work (§\ref{['sec:incorporating']}).
• Figure 3: Dividing the Metric-X scores into equal width buckets can increase the group-by-item correlation by a large margin. However, at the same time, the number of groups used in the correlation (with non-NaN scores) decreases, meaning the corresponding correlations are not fairly comparable since they are computed on different sets of data.
• Figure 4: The generalization of the selected $\epsilon^*$ (dashed line) across datasets appears to depend on specific properties of the datasets. We suspect if the number of ties in the datasets is very different (as in zh-en), the $\epsilon$ is less likely to generalize well.
• Figure 5: The distribution of average pair scores where ties are introduced for Metric-X on WMT'22 zh-en using $\epsilon^*$ as the tie threshold is skewed right with respect to the distribution of all pairs, suggesting the $\epsilon$ is biased toward introducing ties to predict perfect translations.