ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

TL;DR

The paper tackles the mismatch between human feedback and conventional pairwise preference methods by introducing point-wise direct policy optimization (point-wise DPO) and a unified framework, ULMA, that learns from both demonstrations and point-wise preferences in a single step. It formalizes learning from demonstrations and from point-wise preferences, derives a point-wise extension of DPO with decoupled gradients for positive/negative samples, and develops ULMA as a hybrid objective combining SFT on high-quality demonstrations with KL-regularized learning on noisy negatives. Extensive experiments on HH, QA-feedback, red-team, and a high-quality Golden HH dataset show that ULMA consistently outperforms RLHF and DPO, with especially large gains when positive data quality is high. The work demonstrates that unifying demonstration data and point-wise preference data into one objective yields better alignment in diverse settings and offers practical benefits for scalable, robust human-aligned LLM behavior.

Abstract

Aligning language models to human expectations, e.g., being helpful and harmless, has become a pressing challenge for large language models. A typical alignment procedure consists of supervised fine-tuning and preference learning. Most preference learning methods, such as RLHF and DPO, depend on pairwise preference data, which inadequately address scenarios where human feedback is point-wise, leading to potential information loss and suboptimal performance. Addressing this gap, we introduce Point-wise Direct Preference Optimization, a novel preference learning method designed to harness point-wise feedback effectively. Our work also uncovers a novel connection between supervised fine-tuning and point-wise preference learning, culminating in Unified Language Model Alignment, a single-step method that unifies the alignment with human demonstrations and point-wise preferences. Extensive experiments on point-wise preference datasets with binary or continuous labels validate the effectiveness of our methods. Our code and a new dataset with high-quality demonstration samples on harmlessness are released.

Figures (6)

• Figure 1: Win/tie/loss rates on Golden HH. The results on other datasets are given in Appendix \ref{['win_tie_lose_rate']}.
• Figure 2: Performance comparison of different methods on the QA-feedback dataset. The perplexity and the helpful score (in win& tie rate) are reported.
• Figure 2: Performance comparison of various methods on datasets with different levels of quality. Note that each dataset is constructed by replacing a given percentage of low-quality data in HH by high-quality data in Golden HH. In the right plot, the axis of the perplexity metric is reversed.
• Figure 3: Perplexity and harmless of ULMA with varying regularization strength $\beta$ on HH. The axis of the perplexity metric is reversed.
• Figure 4: Performance comparison of various methods in terms of win, tie, and loss rates. From left to right: harmless score on HH, helpful score on QA-feedback, harmless score on red-team.