wDPO: Winsorized Direct Preference Optimization for Robust LLM Alignment

TL;DR

It is shown that robust preference alignment benefits from addressing different noise types with targeted interventions rather than uniform regularization, and is proposed winsorized Direct Preference Optimization (wDPO), a robust LLM alignment approach with hierarchical winsorization.

Abstract

Direct Preference Optimization (DPO) aligns large language models by optimizing pairwise preferences and has shown remarkable effectiveness as a simple and scalable alternative to RLHF. However, in practice, preference data are often noisy. Existing robust variants of DPO mainly rely on uniform objective modifications or global reweighting. While partially effective, these methods treat noisy samples as a homogeneous source of uncertainty and fail to distinguish between different noise types, leading to sub-optimal alignment robustness. In this work, we show that robust preference alignment benefits from addressing different noise types with targeted interventions rather than uniform regularization. We propose winsorized Direct Preference Optimization~(wDPO), a robust LLM alignment approach with hierarchical winsorization. Specifically, wDPO adopts a reward-free hierarchical intervention strategy that leverages only signals already available during DPO training. It first uses the implicit margin from DPO log-ratio to identify heterogeneous noise patterns without relying on external reward models. For hard noise, wDPO performs a data-level intervention by sparsely correcting strongly inconsistent preference pairs. For ambiguous comparisons, it applies a gradient-level intervention through soft winsorization, capping extreme losses in the high-loss tail to prevent weakly informative samples from dominating gradient updates. Extensive experiments on PKU-SafeRLHF and multiple external safety benchmarks demonstrate that wDPO consistently improves preference alignment quality and robustness over vanilla DPO and strong DPO-family baselines, with particularly pronounced gains under controlled label-flip noise.

Figures (5)

• Figure 1: Left: Reward-score gaps in a preference dataset. Reversed pairs exhibit negative gaps, while a large fraction of samples cluster near zero, corresponding to ambiguous comparisons. Right: Under standard DPO, all preference pairs are treated uniformly, causing a small subset of hard or ambiguous pairs to dominate the batch loss and gradient energy, which slows and destabilizes preference separation. wDPO introduces a hierarchical intervention: it sparsely corrects strongly inconsistent pairs and applies soft winsorization to cap the high-loss tail, thereby achieving stable and higher margin compared with DPO.
• Figure 2: Training dynamics of DPO and wDPO on Pythia-2.8B. Top row (a–c): DPO trained for 3 epochs. Bottom row (d–f): wDPO trained for 1 epoch. (a,d) Top Group energy share, measured as the gradient-energy fraction contributed by a fixed top group of samples. (b,e) Global energy uniformity measured by the Herfindahl–Hirschman Index (HHI). (c,f) Preference separation during training, shown by the DPO margin and preference accuracy.
• Figure 3: Hyperparameter sensitivity on Pythia-2.8B, evaluated on PKU-SafeRLHF-30K (test split). Each panel varies one hyperparameter while keeping all others fixed to the default settings in the paper: (a) flip cap $\rho_f$, (b) maximum tail cap $\rho_w^{\max}$, and (c) tail quantile $q$.
• Figure 4: Ablation results with Pythia-2.8B. Bars show WR on the PKU-SafeRLHF-30K test split. The line shows average ASR across five benchmarks, measured by the two judges. We compare standard DPO, wDPO with only Stage I (w/ SI), only Stage II (w/ SII), and the full two-stage wDPO (full).
• Figure 5: Sensitivity to the flip warmup ratio $\alpha$ on Pythia-2.8B. WR is measured on the PKU-SafeRLHF test set. ASR is averaged over five benchmarks.