Learning a Canonical Basis of Human Preferences from Binary Ratings

TL;DR

The paper investigates what preferences are encoded in binary human feedback used for aligning large language models. It develops a pipeline that converts binary choices into granular preferences and topics, clusters them, and refines them into a canonical subset of 21 preferences and 21 topics that capture the majority of variation. The authors validate this canonical basis with both synthetic and empirical experiments, and demonstrate its utility for model evaluation via preference Elo (pElo) and for targeted fine-tuning using LoRA with Direct Preference Optimization to improve alignment. This work enables more interpretable, topic-aware alignment and lays groundwork for personalization by modeling user-specific preference bases.

Abstract

Recent advances in generative AI have been driven by alignment techniques such as reinforcement learning from human feedback (RLHF). RLHF and related techniques typically involve constructing a dataset of binary or ranked choice human preferences and subsequently fine-tuning models to align with these preferences. This paper shifts the focus to understanding the preferences encoded in such datasets and identifying common human preferences. We find that a small subset of 21 preference categories (selected from a set of nearly 5,000 distinct preferences) captures >89% of preference variation across individuals. This small set of preferences is analogous to a canonical basis of human preferences, similar to established findings that characterize human variation in psychology or facial recognition studies. Through both synthetic and empirical evaluations, we confirm that our low-rank, canonical set of human preferences generalizes across the entire dataset and within specific topics. We further demonstrate our preference basis' utility in model evaluation, where our preference categories offer deeper insights into model alignment, and in model training, where we show that fine-tuning on preference-defined subsets successfully aligns the model accordingly.

Figures (11)

• Figure 1: Our pipeline converts a binary rating into a set of common human preferences. (A) This process is run in parallel for each binary choice. (B) This results in close to 5,000 preferences and over 3,000 topics. These preferences and topics are aggregated and then refined, resulting in just 21 preferences and 21 topics covering >89% of the original dataset.
• Figure 2: Word clouds showing underlying, granular preferences.
• Figure 3: Probability ratios as described in \ref{['subsec:eval_methods']}. Comparison using GPT-4o hurst2024gpt, Gemini team2023gemini, and Claude 3.7 anthropic_claude_3_7_sonnet_2025. A ratio of 1 would indicate no preference for the generated or category-specific preference. A ratio > 1 indicates preference for the generated or category-specific preference.
• Figure 4: Prompt for extracting preferences and topics from binary comparison data.
• Figure 5: Prompt for refining preferences.