Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HPS: Hard Preference Sampling for Human Preference Alignment

Xiandong Zou, Wanyu Lin, Yuchen Li, Pan Zhou

Abstract

Aligning Large Language Model (LLM) responses with human preferences is vital for building safe and controllable AI systems. While preference optimization methods based on Plackett-Luce (PL) and Bradley-Terry (BT) models have shown promise, they face challenges such as poor handling of harmful content, inefficient use of dispreferred responses, and, specifically for PL, high computational costs. To address these issues, we propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment. HPS introduces a training loss that prioritizes the most preferred response while rejecting all dispreferred and harmful ones. It emphasizes "hard" dispreferred responses -- those closely resembling preferred ones -- to enhance the model's rejection capabilities. By leveraging a single-sample Monte Carlo sampling strategy, HPS reduces computational overhead while maintaining alignment quality. Theoretically, HPS improves sample efficiency over existing PL methods and maximizes the reward margin between preferred and dispreferred responses, ensuring clearer distinctions. Experiments on HH-RLHF and PKU-Safety datasets validate HPS's effectiveness, achieving comparable BLEU and reward scores while greatly improving reward margins and thus reducing harmful content generation.

HPS: Hard Preference Sampling for Human Preference Alignment

Abstract

Aligning Large Language Model (LLM) responses with human preferences is vital for building safe and controllable AI systems. While preference optimization methods based on Plackett-Luce (PL) and Bradley-Terry (BT) models have shown promise, they face challenges such as poor handling of harmful content, inefficient use of dispreferred responses, and, specifically for PL, high computational costs. To address these issues, we propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment. HPS introduces a training loss that prioritizes the most preferred response while rejecting all dispreferred and harmful ones. It emphasizes "hard" dispreferred responses -- those closely resembling preferred ones -- to enhance the model's rejection capabilities. By leveraging a single-sample Monte Carlo sampling strategy, HPS reduces computational overhead while maintaining alignment quality. Theoretically, HPS improves sample efficiency over existing PL methods and maximizes the reward margin between preferred and dispreferred responses, ensuring clearer distinctions. Experiments on HH-RLHF and PKU-Safety datasets validate HPS's effectiveness, achieving comparable BLEU and reward scores while greatly improving reward margins and thus reducing harmful content generation.

Paper Structure

This paper contains 43 sections, 6 theorems, 87 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Methodology
  5. Motivation: Analysis of PL & BT Training Losses
  6. Hard Preference Sampling for Alignment
  7. Theoretical Analysis
  8. Sample Complexity Analysis
  9. Reward Margin Analysis
  10. Experiments
  11. Fine-Tuning Setting
  12. Transfer Learning Setting
  13. Ablation Study
  14. Conclusion
  15. Reinforcement Learning from Human Feedback
  16. ...and 28 more sections

Key Result

Theorem 1

With Assumption assum1, with probability at least $1-\delta$, the distance between the optimum solution $\boldsymbol{\theta}_{\text{HPS}}$ of our HPS loss and the ground-truth optimum $\boldsymbol{\theta}^{*}$ can be bounded: where $\zeta \!=\!\frac{1}{2+\exp\left(2\alpha_{0}+\ln(n-1)\right)+\exp\left(-2\alpha_{0}\right)}$ and $\Sigma_{\mathcal{D}}\!=\!\frac{2}{mn(n-1)}$$\sum\limits_{i=1}^{m}\!\s

Figures (1)

  • Figure 1: Example for harmless and preferred response $y_{\tau(1)}$ and harmful and dispreferred response $y_{\tau(2)}$ and $y_{\tau(3)}$. $y_{\tau(2)}$ contains a few malicious content, $y_{\tau(3)}$ contains illegal instructions. Harmful content is highlighted with underlining.

Theorems & Definitions (9)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem
  • proof
  • Theorem
  • proof
  • Theorem
  • proof