Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics

TL;DR

This study reveals systematic length bias in both regression-based and LLM-based QE metrics, where longer translations are unfairly penalized and shorter hypotheses are favored when multiple candidates exist. It formalizes the bias with Bias(hat{theta}) = E[hat{theta}] - b8 and demonstrates that current QE systems exhibit length sensitivity across 10 language pairs. The authors propose two mitigation strategies: (i) a training-time length-normalization objective, and (ii) incorporating reference texts during evaluation, both of which reduce length bias and improve cross-length fairness. The findings have practical implications for QE-based reranking and QE-guided reinforcement learning, urging length-robust evaluation practices and more representative training data to ensure reliable, fair quality judgments across translations of varying lengths.

Abstract

Quality Estimation (QE) metrics are vital in machine translation for reference-free evaluation and as a reward signal in tasks like reinforcement learning. However, the prevalence and impact of length bias in QE have been underexplored. Through a systematic study of top-performing regression-based and LLM-as-a-Judge QE metrics across 10 diverse language pairs, we reveal two critical length biases: First, QE metrics consistently over-predict errors with increasing translation length, even for high-quality, error-free texts. Second, they exhibit a preference for shorter translations when multiple candidates are available for the same source text. These inherent length biases risk unfairly penalizing longer, correct translations and can lead to sub-optimal decision-making in applications such as QE reranking and QE guided reinforcement learning. To mitigate this, we propose two strategies: (a) applying length normalization during model training, and (b) incorporating reference texts during evaluation. Both approaches were found to effectively reduce the identified length bias.

Figures (13)

• Figure 2: Trends of metric scores with increasing source text length across different metrics. Each data point denotes the score change relative to the first passage.
• Figure 3: Trends of MetricX-24 QE score with different severity/error types, evaluated on WMT24++ En-De and En-Zh.
• Figure 4: Trends of Gemini AutoMQM score with different severity/error types, evaluated on WMT24++ En-De and En-Zh.
• Figure 5: Average preference for shorter translations by length difference.
• Figure 6: Distribution of MetricX QE outputs and gold standard scores.