Beyond Single-Reward: Multi-Pair, Multi-Perspective Preference Optimization for Machine Translation

TL;DR

This work tackles the problem of aligning MT models to human preferences by addressing two core issues: unreliable reward signals from Quality Estimation (QE) and inefficient use of candidate translations. It introduces M2PO, a framework that combines a multi-perspective reward engine (a dedicated hallucination penalty and a dynamic scoring curriculum) with a multi-pair optimization strategy that exhaustively leverages the full candidate pool. The static reward $r_s$ blends QE quality with a factuality bonus via $r_s = r_{qe} + \lambda_f S_{align}$ and is fused online with the model's judgments to produce $r_{fused}$, guided by a curriculum $\alpha_t$. The optimization uses a composite loss $\mathcal{L}_{\text{M2PO}} = \lambda_{\text{pref}}\mathcal{L}_{\text{DM-DPO}} + \lambda_{\text{rank}}\mathcal{L}_{\text{Rank}} + \lambda_{\text{bc}}\mathcal{L}_{\text{BC}}$, with multi-pair construction, a dynamic weighting in the DM-DPO loss, and a ListNet-style stability anchor. Empirically, M2PO achieves state-of-the-art results among open-source MT systems and is highly competitive with proprietary LLMs on the WMT21-22 benchmarks, delivering improvements in both translation quality and faithfulness, and showing strong generalization across DPO-like algorithms.

Abstract

Direct Preference Optimization (DPO) is a powerful paradigm for aligning Large Language Models (LLMs) to human preferences in Machine Translation (MT), but current methods are hindered by two fundamental challenges: (1) flawed reward signals from Quality Estimation (QE) models that overlook critical errors like translation hallucination, and (2) inefficient data utilization that discards valuable learning signals by selecting only a single win-loss pair. To address these limitations, we introduce M^2PO: Multi-Pair, Multi-Perspective Preference Optimization. Our framework integrates a multi-perspective reward engine that creates a more robust signal by combining two key viewpoints: a new hallucination penalty for factuality, and an innovative dynamic quality score that adaptively fuses external evaluations with the model's own evolving judgment. This is synergistically paired with a multi-pair construction strategy that systematically creates a comprehensive set of preference pairs from the entire pool of translation candidates. This synergistic approach ensures the model learns from a richer spectrum of quality trade-offs, leading to more robust and faithful translations. On challenging WMT21-22 benchmarks, M^2PO substantially outperforms existing preference optimization methods and demonstrates highly competitive performance against leading proprietary LLMs.

Figures (4)

• Figure 1: The relationship between the XCOMET score and our factuality-focused Coverage Score. The plot reveals significant volatility in the mid-coverage range (approximately 30-70). This is precisely the region where partial hallucinations and omissions are most prevalent.
• Figure 2: The M2PO framework. Stage 1: Candidate Set Construction creates a diverse dataset. Stage 2: Dynamic Preference Modeling computes a robust static score ($r_s$) offline by combining QE with a hallucination penalty, then fuses it with the model's own judgment ($r_d$) to create an online, dynamic fused score ($r_{\text{fused}}$) for ranking. Stage 3: Multi-Component Optimization uses this ranking to form multiple preference pairs and trains the model with a composite loss.
• Figure 3: Translation Quality (XCOMET) vs. Faithfulness (Coverage Score). Average XCOMET Score and Average Coverage Score mean average all 10 translation directions score.
• Figure 4: An overview of the prompts employed in our study. This includes the prompt for faithfulness evaluation (Coverage Calculation), the modified prompt for GPT-4o/mini, and the prompt for our M2PO model.