Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, Asuman Ozdaglar

TL;DR

This work addresses the challenge of heterogeneous human preferences in RLHF by introducing two principled frameworks: personalization-based RLHF, which learns either per-user reward models via a shared representation or clusters users into groups, and aggregation-based RLHF, which combines diverse rewards or probabilistic opinions under formal axioms and mechanism design. It provides theoretical guarantees on sample complexity, algorithmic convergence, and sub-optimality gaps under assumptions like task diversity and representation uniqueness, plus a lower-bound perspective. It also develops a probabilistic opinion pooling approach that links preference aggregation directly to reward aggregation under the Plackett–Luce model, and designs a DSIC mechanism to incentivize truthful reporting with welfare maximization. Empirical results on a text-summarization task validate the practical benefits of personalization and probabilistic-opinion aggregation, highlighting improved reward modeling and robust aggregation behavior across multiple datasets and model families.

Abstract

Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

TL;DR

This work addresses the challenge of heterogeneous human preferences in RLHF by introducing two principled frameworks: personalization-based RLHF, which learns either per-user reward models via a shared representation or clusters users into groups, and aggregation-based RLHF, which combines diverse rewards or probabilistic opinions under formal axioms and mechanism design. It provides theoretical guarantees on sample complexity, algorithmic convergence, and sub-optimality gaps under assumptions like task diversity and representation uniqueness, plus a lower-bound perspective. It also develops a probabilistic opinion pooling approach that links preference aggregation directly to reward aggregation under the Plackett–Luce model, and designs a DSIC mechanism to incentivize truthful reporting with welfare maximization. Empirical results on a text-summarization task validate the practical benefits of personalization and probabilistic-opinion aggregation, highlighting improved reward modeling and robust aggregation behavior across multiple datasets and model families.

Abstract

Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.
Paper Structure (61 sections, 30 theorems, 103 equations, 2 figures, 1 table, 7 algorithms)

This paper contains 61 sections, 30 theorems, 103 equations, 2 figures, 1 table, 7 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Reinforcement Learning from Human Feedback.
  4. Representation Learning.
  5. Reward and Preference Aggregation.
  6. Comparison with Recent Works.
  7. Fundamentals of Auction Theory.
  8. Notation.
  9. Preliminaries
  10. Markov Decision Processes.
  11. Relationship between Preference and Reward Function.
  12. Provable Personalized RLHF via Representation Learning
  13. Learning Personalized Reward Model
  14. Reward Function Class.
  15. Algorithms
  16. ...and 46 more sections

Key Result

Corollary 3.1

(Closeness between $\psi^\star$ and $\psi_{\omega}$). Suppose Assumptions assum:real, assum:task_diverse, and assum:psi-unique hold. For any $\delta \in (0,1]$, with probability at least $1-\delta$, if $\bm{r}_{\omega, \pmb{\theta}} \in \mathcal{R}'(\mathcal{D})$ as specified in alg:personal, then t for all $\tau_0, \tau_1$, where $c_{\text{rep}} >0$ is a constant.

Figures (2)

  • Figure 1: We demonstrate a setting where humans might have heterogeneous feedback. We provide a personalization-based framework and a human preference aggregation-based framework.
  • Figure 2: Accuracy of different methods with 3 times experiments. P: Personalized, C: Clustered, G(L): General (Linear) representation. Naive RLHF: original training method.

Theorems & Definitions (43)

  • Definition 3.1: Concentrability Coefficient
  • Definition 3.2
  • Corollary 3.1
  • Theorem 3.1
  • Theorem 3.2
  • Remark 1: Sample Complexity
  • Theorem 3.3
  • Definition 3.3: Label Discrepancy
  • Lemma 1: mansour2020three
  • Theorem 3.4
  • ...and 33 more