Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior

TL;DR

This work tackles how to operationalize value pluralism in LLM alignment by systematically varying both who provides alignment feedback (demographic groups) and how that feedback is used (rating scales, aggregation, and optimization methods). Using a bilingual English–German pipeline and $N=1{,}095$ participants with $27{,}375$ ratings across five alignment dimensions, the authors show that demographic signals produce dimension-specific effects, and that technical design choices—particularly preserving rater disagreement, employing 5-point scales, and using Direct Preference Optimization (DPO)—consistently improve safety and social understanding relative to conventional approaches. The results reveal that misalignment can arise from both data composition and methodological choices, with DPO outperforming GRPO and single-objective tuning sometimes surpassing multi-objective training. The study argues for robust, pluralistic alignment practices that preserve minority safety perspectives and continuously audit who dominates the data and which design decisions amplify or suppress diverse values, contributing to safer and more inclusive AI systems in practice.

Abstract

Although large language models (LLMs) are increasingly trained using human feedback for safety and alignment with human values, alignment decisions often overlook human social diversity. This study examines how incorporating pluralistic values affects LLM behavior by systematically evaluating demographic variation and design parameters in the alignment pipeline. We collect alignment data from US and German participants (N = 1,095 participants, 27,375 ratings) who rated LLM responses across five dimensions: Toxicity, Emotional Awareness (EA), Sensitivity, Stereotypical Bias, and Helpfulness. We fine-tuned multiple Large Language Models and Large Reasoning Models using preferences from different social groups while varying rating scales, disagreement handling methods, and optimization techniques. The results revealed systematic demographic effects: male participants rated responses 18% less toxic than female participants; conservative and Black participants rated responses 27.9% and 44% higher on EA than liberal and White participants, respectively. Models fine-tuned on group-specific preferences exhibited distinct behaviors. Technical design choices showed strong effects: the preservation of rater disagreement achieved roughly 53% greater toxicity reduction than majority voting, and 5-point scales yielded about 22% more reduction than binary formats; and Direct Preference Optimization (DPO) consistently outperformed Group Relative Policy Optimization (GRPO) in multi-value optimization. These findings represent a preliminary step in answering a critical question: How should alignment balance expert-driven and user-driven signals to ensure both safety and fair representation?

Figures (10)

• Figure 1: Conditional probabilities between alignment dimensions. Each cell shows the probability of a positive rating in the column given a positive rating in the row.
• Figure 2: Effects of demographic composition on fine-tuning outcomes. Purple circles show model differences between groups (Liberal, Female, White minus Conservative, Male, Black, respectively) on Emotional Awareness or Toxicity ratings; blue triangles show survey-estimated differences for the same contrasts. Positive values indicate higher Emotional Awareness for the first group; negative values indicate lower Toxicity for the first group. Error bars show 95% confidence intervals.
• Figure 3: Rating Scale Effects on Toxicity Reduction. (A) All scales reduce toxicity relative to the control (no fine-tuning), with the 5-point most effective. (B) 5-point scales significantly outperform binary. Error bars: 95% CIs. Lower values indicate reduced toxicity.
• Figure 4: Disagreement Handling Strategy Effects on Alignment Training. (A) Strategy performance relative to the control (no fine-tuning) on toxicity. (B) Pairwise strategy comparisons. Error bars show 95% CIs. Lower values indicate better performance.
• Figure 5: DPO and GRPO Optimization Methods Comparison. (A-B) Performance of each DPO and GRPO trained model. (C-D) Pairwise comparisons between single-objective and multi-objective DPO approaches. Error bars show 95% CIs.