CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences

TL;DR

Confidence-Aware Preference Optimization (CAPO) is proposed, which replaces DPO's fixed treatment of preference pairs with a dynamic loss scaling mechanism based on a relative reward, and improves alignment by widening the gap between preferred and dispreferred responses across languages.

Abstract

Preference optimization is a critical post-training technique used to align large language models (LLMs) with human preferences, typically by fine-tuning on ranked response pairs. While methods like Direct Preference Optimization (DPO) have proven effective in English, they often fail to generalize robustly to multilingual settings. We propose a simple yet effective alternative, Confidence-Aware Preference Optimization (CAPO), which replaces DPO's fixed treatment of preference pairs with a dynamic loss scaling mechanism based on a relative reward. By modulating the learning signal according to the confidence in each preference pair, CAPO enhances robustness to noisy or low-margin comparisons, typically encountered in multilingual text. Empirically, CAPO outperforms existing preference optimization baselines by at least 16% in reward accuracy, and improves alignment by widening the gap between preferred and dispreferred responses across languages.

Figures (13)

• Figure 1: Density distribution of reward differences ($\Delta r$) across languages after DPO (top) and after applying our proposed confidence-aware preference optimization - CAPO (bottom). CAPO shifts the distributions towards higher $\Delta r$ values, indicating improved separation between preferred and dispreferred responses.
• Figure 2: An example of a preference pair where both responses appear similarly plausible. CAPO leverages RRM to interpret such cases and provide a more informative learning signal. In other words, RRM boosts confidence in favor of the preferred response.
• Figure 3: Shift in reward signal across it and de under llama and DPO. Although there is a shift of reward difference towards the right, there are a lot of samples for which the difference is negative.
• Figure 4: Training Loss vs. Steps for DPO and CAPO: CAPO demonstrates improved stability and convergence over DPO and DPONLL.
• Figure 5: Comparison of reward accuracy between DPO and CAPO on the validation data. CAPO shows improved accuracy across all languages except in en.