No Preference Left Behind: Group Distributional Preference Optimization

TL;DR

This work tackles the problem that language models often align to a single dominant set of human preferences, neglecting intra-group diversity. It introduces Group Distributional Preference Optimization (GDPO), a belief-based framework that first calibrates the group’s belief distribution $p^*_{\mathcal{B}}$ and then performs belief-conditioned preference alignment, trained from a supervised fine-tuned checkpoint. Empirically, GDPO outperforms Direct Preference Optimization (DPO) on synthetic controllable opinion generation and real-world controllable movie review generation, achieving better distributional alignment and producing minority beliefs at test time. The approach advances pluralistic alignment by explicitly modeling beliefs as latent variables and conditioning responses on them, with potential impact on policy simulation and inclusive AI systems.

Abstract

Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distributional Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences. GDPO calibrates a language model using statistical estimation of the group's belief distribution and aligns the model with belief-conditioned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movie review datasets, we show that DPO fails to align with the targeted belief distributions, while GDPO consistently reduces this alignment gap during training. Moreover, our evaluation metrics demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advance in pluralistic alignment.

Figures (5)

• Figure 1: Demonstration of GDPO. Training Dataset: We create the belief-conditioned preference datasets for training, where people's beliefs on a topic are diverse according to a specified distribution, and their preferences are conditioned on those beliefs. Training Objective: Instead of optimizing all preferences simultaneously, GDPO first calibrates the belief distribution, followed by belief-conditioned preference alignment. Inference Time: When a new query is received, the model predicts a belief and generates responses based on it.
• Figure 2: Reward Margins During DPO Training: Majority/Minority means the chosen response $y_c$ is from majority/minority preferences in the dataset.
• Figure 3: Reward Margins During GDPO Training: Majority/Minority means the chosen response $y_c$ is from majority/minority preferences in the evaluation dataset.
• Figure 4: Avg. JSD During the Training Process: The dash lines show distributions without any training; the solid lines represent methods having the training process.
• Figure 5: Evaluation of Controllable Review Generation

Theorems & Definitions (1)

• Definition 4.1: Human Belief