Small-Margin Preferences Still Matter-If You Train Them Right

TL;DR

This work addresses how small-margin, ambiguous preference pairs affect LLM alignment and challenges the default practice of discarding them. It introduces MixDPO, a difficulty-aware training method that orders data by the margin-based difficulty $M(s_w,s_l)$ and applies a hybrid loss: $\mathcal{L}_{\text{MixDPO}} = -(1 - z)\log p_\theta(y_w \succ y_l \mid x) - z \frac{1}{T}\sum_{t=1}^{T} \log p_\theta(y_{w,t} \mid x, y_{w,<t})$ with $z=1$ if $M(s_w,s_l) < \tau$, thereby leveraging easy pairs with DPO and difficult ones with SFT. Across three open-ended benchmarks, including AlpacaEval 2 LC, MixDPO consistently outperforms DPO and several variants, indicating a practical, data-centric path to improved alignment. The results demonstrate that difficulty-aware data utilization can harness informative signals from ambiguous pairs without incurring the instability typical of purely preference-based optimization. Limitations include reliance on rating-derived difficulty and potential score noise, suggesting future work on alternative difficulty metrics and score curation strategies.

Abstract

Preference optimization methods such as DPO align large language models (LLMs) using paired comparisons, but their effectiveness can be highly sensitive to the quality and difficulty of preference pairs. A common heuristic treats small-margin (ambiguous) pairs as noisy and filters them out. In this paper, we revisit this assumption and show that pair difficulty interacts strongly with the optimization objective: when trained with preference-based losses, difficult pairs can destabilize training and harm alignment, yet these same pairs still contain useful supervision signals when optimized with supervised fine-tuning (SFT). Motivated by this observation, we propose MixDPO, a simple yet effective difficulty-aware training strategy that (i) orders preference data from easy to hard (a curriculum over margin-defined difficulty), and (ii) routes difficult pairs to an SFT objective while applying a preference loss to easy pairs. This hybrid design provides a practical mechanism to leverage ambiguous pairs without incurring the optimization failures often associated with preference losses on low-margin data. Across three LLM-judge benchmarks, MixDPO consistently improves alignment over DPO and a range of widely-used variants, with particularly strong gains on AlpacaEval~2 length-controlled (LC) win rate.

Figures (13)

• Figure 1: Performance comparison between random ordering and difficulty-based sorting on the AlpacaEval 2 benchmark. DPO is used as the default loss function. Notably, instead of discarding difficult preference pairs, further training them using the SFT loss leads to improved performance.
• Figure 2: Original rating score margin distribution. Left: Ultrafeedback dataset (61k samples). Right: Argilla dataset (7k samples). Chosen/Rejected scores are both annotated from various LLMs. Observe that approximately 50% of the samples in both datasets exhibit a score difference below 1.0.
• Figure 3: Alpaca LC-WR
• Figure 4: Train loss
• Figure 5: Reward accuracy

Theorems & Definitions (1)

• Definition 3.1