CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation

TL;DR

CRPO tackles data-selection bottlenecks in offline preference optimization for machine translation by jointly leveraging reward signals and model confidence to pick informative sentence pairs. It introduces two variants, CR$+$ and CR$\times$, derived from loss change and loss value respectively, and demonstrates that a CR-Score-guided selection yields superior translation quality and data efficiency across ten directions and for encoder-decoder models like NLLB. Empirically, CRPO outperforms RS-DPO, RSO, MBR score, and Triplet data on COMET/BLEURT metrics, while ablations confirm the necessity of combining reward and confidence. The approach generalizes to multilingual contexts and offers a scalable path for robust MT fine-tuning without the full RLHF complexity.

Abstract

Large language models (LLMs) have shown great potential in natural language processing tasks, but their application to machine translation (MT) remains challenging due to pretraining on English-centric data and the complexity of reinforcement learning from human feedback (RLHF). Direct Preference Optimization (DPO) has emerged as a simpler and more efficient alternative, but its performance depends heavily on the quality of preference data. To address this, we propose Confidence-Reward driven Preference Optimization (CRPO), a novel method that combines reward scores with model confidence to improve data selection for fine-tuning. CRPO selects challenging sentence pairs where the model is uncertain or underperforms, leading to more effective learning. While primarily designed for LLMs, CRPO also generalizes to encoder-decoder models like NLLB, demonstrating its versatility. Empirical results show that CRPO outperforms existing methods such as RS-DPO, RSO and MBR score in both translation accuracy and data efficiency.

Figures (4)

• Figure 1: Comparison among RSO, RS-DPO and CRPO. RSO scores candidate responses with reward and applies statistical reject sampling for preference dataset. RS-DPO accepts sentence pairs with reward difference surpass a preset threshold. Instead, proposed CRPO evaluates sentence pairs with both reward different and policy confidence.
• Figure 2: Visualization of reward score and $\log\pi_{ref}$ for the entire candidate dataset, as well as preferred and dis-preferred sentences selected by CRPO$+$, CRPO$\times$, RS-DPO, RSO and MBR-BW.
• Figure 3: Reward score vs. $\log\pi_{ref}$ for entire dataset.
• Figure 4: Distribution of reward scores and $\log\pi_{ref}$.