Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
Keertana Chidambaram, Karthik Vinary Seetharaman, Vasilis Syrgkanis
TL;DR
This paper tackles unobserved preference heterogeneity in aligning large language models by extending Direct Preference Optimization (DPO) to handle diverse annotators. It introduces EM-DPO to jointly cluster annotators and train type-specific policy ensembles, and MMRA to fairly aggregate these ensembles under a minimax regret criterion. The authors prove identifiability results showing that binary preferences are insufficient for learning heterogeneous preferences, while ternary (three-item) or richer multi-item feedback enables unique recovery of latent distributions under reasonable conditions. Empirically, EM-DPO with ternary feedback improves reward margins and, when combined with MMRA, provides robust, fair performance across diverse user groups on GlobalOpinionQA and MPI datasets. The work advances pluralistic alignment by enabling personalization and fairness without explicit reward modeling, while outlining limitations and directions for extending identifiability and scalability to continuous preference spaces.
Abstract
Reinforcement Learning from Human Feedback (RLHF) has become central to aligning large language models with human values, typically by first learning a reward model from preference data which is then used to update the model with reinforcement learning. Recent alternatives such as Direct Preference Optimization (DPO) simplify this pipeline by directly optimizing on preferences. However, both approaches often assume uniform annotator preferences and rely on binary comparisons, overlooking two key limitations: the diversity of human evaluators and the limitations of pairwise feedback. In this work, we address both these issues. First, we connect preference learning in RLHF with the econometrics literature and show that binary comparisons are insufficient for identifying latent user preferences from finite user data and infinite users, while (even incomplete) rankings over three or more responses ensure identifiability. Second, we introduce methods to incorporate heterogeneous preferences into alignment algorithms. We develop an Expectation-Maximization adaptation of DPO that discovers latent annotator types and trains a mixture of LLMs accordingly. Then we propose an aggregation algorithm using a min-max regret fairness criterion to produce a single generative policy with equitable performance guarantees. Together, these contributions establish a theoretical and algorithmic framework for fairness and personalization for diverse users in generative model alignment.