Hummer: Towards Limited Competitive Preference Dataset

TL;DR

This work addresses the problem that RLHF preference datasets often encode conflicting alignment objectives, weakening safety and limiting domain-specific fine-tuning. It introduces Alignment Dimension Conflict (ADC) as a statistical metric and presents Hummer, a low-conflict preference dataset built from UltraFeedback with GPT-4 annotation, plus a fine-grained variant Hummer-F that filters noisy pairs. The authors train reward models (HummerRM and HummerRM-F) with a hybrid sampling strategy that adaptively balances alignment dimensions, achieving substantially lower ADCs and stronger RewardBench performance than baselines, while exhibiting improved resistance to jailbreak attacks. The findings suggest that targeted, high-quality, low-conflict preference data can enable safer, more tunable downstream customization in RLHF systems, with potential for unsupervised discovery of low-conflict objectives in future work.

Abstract

Preference datasets are essential for incorporating human preferences into pre-trained language models, playing a key role in the success of Reinforcement Learning from Human Feedback. However, these datasets often demonstrate conflicting alignment objectives, leading to increased vulnerability to jailbreak attacks and challenges in adapting downstream tasks to prioritize specific alignment objectives without negatively impacting others. In this work, we introduce a novel statistical metric, Alignment Dimension Conflict, to quantify the degree of conflict within preference datasets. We then present \texttt{Hummer} and its fine-grained variant, \texttt{Hummer-F}, as innovative pairwise preference datasets with reduced-conflict alignment objectives. \texttt{Hummer} is built based on UltraFeedback and is enhanced by AI feedback from GPT-4, marking as the first preference dataset aimed at reducing the competition between alignment objectives. Furthermore, we develop reward models, HummerRM and HummerRM-F, which employ a hybrid sampling approach to balance diverse alignment objectives effectively. This sampling method positions HummerRM as an ideal model for domain-specific further fine-tuning and reducing vulnerabilities to attacks.

Figures (5)

• Figure 1: The ADC estimation pipeline, measuring the negative performance gap between initial and further fine-tuned reward models.
• Figure 2: (a) Normal distribution of ADC with varying standard variance $\sigma$: $\mathbb{E}_{x \sim \mathcal{N}\left(0, \sigma^2\right)}\textrm{U}[x]$. (b-c) The performance deviation with further fine-tuning on the first dimension of preference datasets with (b) low and (c) high ADC. Intuitively, a high ADC indicates a strong conflict between the alignment dimensions of a given preference dataset.
• Figure 3: Hummer construction process. We leverage the advanced ability of GPT-4 to build Hummer, a preference dataset with low competitive alignment objectives.
• Figure 4: The performance deviation with further fine-tuning on different alignment objectives, where the green bar indicates the further fine-tuning dimensions. Notably, Hummer demonstrates minimal competition among alignment dimensions.
• Figure 5: Performance with different sampling strategies on imbalanced datasets.

Theorems & Definitions (2)

• Definition 1: Alignment Dimension Conflict
• Definition 2: Alignment Dimension Conflict Benchmark