MixDPO: Modeling Preference Strength for Pluralistic Alignment

TL;DR

Preference-based alignment often assumes uniform preference strength, which ignores heterogeneity in how strongly humans express judgments. MixDPO generalizes Direct Preference Optimization by treating the preference strength parameter $\beta$ as a random variable drawn from a learned distribution, with LogNormal or Gamma families providing scalable options. Across three diverse datasets and two open-weight language models, MixDPO yields higher macro-level margins and preserves micro-level subgroup preferences, especially where heterogeneity is pronounced, while incurring modest computational overhead. This distributional approach makes preference heterogeneity observable and actionable, offering a principled route toward pluralistic alignment that respects diverse values without sacrificing aggregate performance.

Abstract

Preference based alignment objectives implicitly assume that all human preferences are expressed with equal strength. In practice, however, preference strength varies across individuals and contexts -- a phenomenon established in behavioral economics and discrete choice theory. This mismatch limits the ability of existing objectives to faithfully capture heterogeneous human judgments. Inspired by this literature, we introduce Mixed Logit Direct Preference Optimization (MixDPO), a generalization of Direct Preference Optimization that models variation in preference strength. MixDPO enables alignment objectives to capture heterogeneity in how strongly preferences are expressed across training examples. We evaluate MixDPO on three preference datasets using two open-weight language models. Across datasets, MixDPO improves aggregate alignment performance (+11.2 points on Pythia-2.8B) while preserving subgroup level preferences, with the largest gains appearing in settings with higher inferred preference heterogeneity. MixDPO makes preference heterogeneity explicit through learned strength distributions. We release our code for reproducibility.

Figures (7)

• Figure 1: Motivating example of latent preference strength heterogeneity. Although annotators provide only binary comparisons, variation in repeated preference judgments—holding preference direction fixed—can arise from latent differences in how sharply individuals distinguish between alternatives. Mixed logit models capture this variation through a random sensitivity parameter.
• Figure 2: Education subgroup mean preference margins for PRISM and Community Alignment. Micro averages weight subgroups by frequency and can mask degraded alignment for underrepresented groups; macro averages assign equal weight per subgroup and better reflect subgroup conditioned performance. Marker size indicates subgroup sample size.
• Figure 3: Preference margin gains over DPO on PRISM (Pythia-2.8B). The figure reports subgroup level preference margin gains for demographic subgroups (annotator heterogeneity) and conversational framing (contextual heterogeneity), along with micro and macro averaged margins. MixDPO (LogNormal and Gamma) improve subgroup averaged (macro) preference margins relative to standard DPO while maintaining comparable micro averaged performance. This suggests improved preservation of subgroup preferences without aggregate tradeoffs.
• Figure 4: Training trajectories of the variances of the learned preference strength distribution under Gamma MixDPO (Pythia-2.8B, Llama-1B; $\beta$ learning rate $10^{-4}$). Anthropic HH converges to lower mean and variance, while PRISM and Community Alignment retain higher mean and variance throughout training.
• Figure 5: Relative runtime comparison. MixDPO introduces modest computational overhead: the LogNormal variant adds cost due to Monte Carlo sampling, while the Gamma variant incurs additional overhead from numerical evaluation of closed form expectation.