PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation

TL;DR

PEAR introduces a pairwise, supervised quality estimation framework for MT evaluation that predicts the graded difference between two translations of the same source. It leverages a cross-encoder architecture with a pairwise head and trains on human MQM-derived differences, augmented with an antisymmetry regularization to enforce sign inversion under input-order reversal. On the WMT24 MQM meta-evaluation, PEAR outperforms strictly matched single-candidate QE baselines and even larger QE models, while using far fewer parameters, and it yields a distinct evaluation signal evidenced by lower segment-level correlation with other metrics. PEAR also demonstrates practicality as a reference-anchored mode and as a robust MBR utility, reducing pairwise scoring costs without relying on human references. The work provides released code and checkpoints and outlines promising directions for scaling and targeted data strategies.

Abstract

We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised Quality Estimation (QE) metric family that reframes reference-free Machine Translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for Minimum Bayes Risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.

Figures (5)

• Figure 1: Segment- and system-level PEAR evaluation. For each source segment $s_i$, PEAR compares the system outputs $(mt_i^A, mt_i^B)$ and predicts a relative score $\Delta_i$. The sign indicates which translation is preferred ($\Delta_i>0$: $mt_i^A$; $\Delta_i<0$: $mt_i^B$), while the magnitude $|\Delta_i|$ reflects the strength of the preference, with smaller magnitudes corresponding to weaker preferences approaching a tie. The system-level PEAR score is then the arithmetic average of the segment-level PEAR scores.
• Figure 2: Rank stability of PEAR$_{\mathrm{ref}}$ across anchors. The leftmost anchor (Human Ref) uses the human reference as the fixed comparison translation, matching Table \ref{['tab:wmt24_main_res']}. The remaining anchors use an MT output; for each MT anchor, that system is removed from the benchmark before recomputing meta-evaluation. Ranks are computed by Avg Corr (lower is better).
• Figure 3: En-De Pearson correlation matrix between segment-level pairwise difference scores derived from WMT24 metric scores. Single-candidate metrics are converted into pairwise differences by subtraction, while PEAR produces pairwise scores directly. In this matrix, PEAR corresponds to PEAR$_{\mathrm{both}}$ and PEAR_GPT-4.1_distil corresponds to PEAR$_{\mathrm{both,KD}}$.
• Figure 4: En-Es Pearson correlation matrix between segment-level pairwise difference scores derived from WMT24 metric scores. Single-candidate metrics are converted into pairwise differences by subtraction, while PEAR produces pairwise scores directly. In this matrix, PEAR corresponds to PEAR$_{\mathrm{both}}$ and PEAR_GPT-4.1_distil corresponds to PEAR$_{\mathrm{both,KD}}$.
• Figure 5: Ja-Zh Pearson correlation matrix between segment-level pairwise difference scores derived from WMT24 metric scores. Single-candidate metrics are converted into pairwise differences by subtraction, while PEAR produces pairwise scores directly. In this matrix, PEAR corresponds to PEAR$_{\mathrm{both}}$ and PEAR_GPT-4.1_distil corresponds to PEAR$_{\mathrm{both,KD}}$.