Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, William S. Zwicker

TL;DR

The paper argues that social choice theory offers a principled framework to aggregate diverse human feedback for AI alignment, addressing limitations of RLHF and CAI. It proposes concrete avenues like RLCHF and simulated collective decisions to incorporate collective preferences, while examining relevant concepts such as independence of clones, strategic voting, and anonymity. The authors call for rigorous, interdisciplinary work to define who provides input, how feedback is formatted and processed, and how to harmonize multiple stakeholder perspectives. If developed carefully, this approach could yield fairer, more representative, and more robustly accepted AI systems, though it also raises complex technical and governance challenges.

Abstract

Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, such as helping to commit crimes or producing racist text. One approach to fine-tuning, called reinforcement learning from human feedback, learns from humans' expressed preferences over multiple outputs. Another approach is constitutional AI, in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How can we aggregate the input into consistent data about "collective" preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent workshop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023.

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

TL;DR

The paper argues that social choice theory offers a principled framework to aggregate diverse human feedback for AI alignment, addressing limitations of RLHF and CAI. It proposes concrete avenues like RLCHF and simulated collective decisions to incorporate collective preferences, while examining relevant concepts such as independence of clones, strategic voting, and anonymity. The authors call for rigorous, interdisciplinary work to define who provides input, how feedback is formatted and processed, and how to harmonize multiple stakeholder perspectives. If developed carefully, this approach could yield fairer, more representative, and more robustly accepted AI systems, though it also raises complex technical and governance challenges.

Abstract

Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, such as helping to commit crimes or producing racist text. One approach to fine-tuning, called reinforcement learning from human feedback, learns from humans' expressed preferences over multiple outputs. Another approach is constitutional AI, in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How can we aggregate the input into consistent data about "collective" preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent workshop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023.
Paper Structure (23 sections, 4 figures)

This paper contains 23 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Value Alignment
  4. Reinforcement Learning from Human Feedback
  5. Alternate Preference-Based Fine-Tuning Objectives
  6. Constitutional AI
  7. Social Choice
  8. What Are the Collective Decision Problems and their Alternatives in this Context?
  9. Who Provides the Human Feedback?
  10. What Is the Format of Human Feedback?
  11. Multiple Format Options
  12. Processing Diverse and Informal Feedback
  13. How Do We Incorporate Diverse Individual Feedback?
  14. Proposal: Reinforcement Learning from Collective Human Feedback (RLCHF)
  15. Proposal: Simulated Collective Decisions
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Individual rankings on the left (4 voters say $ABC$, 4 say $ACB$, etc.) lead to different aggregations on the right, depending on the aggregation rule. Borda Count gives an alternative 0, 1, or 2 points for each voter who ranks it last, second, or first, respectively; alternatives are then ordered by score. Instant Runoff ranks $C$ last since $C$ has the fewest first-place rankings; after removing $C$, $B$ has the fewest first-place rankings, so $B$ is in second and $A$ first. For Ranked Pairs, notice there is a majority cycle: a majority of voters prefer $A$ to $B$, a majority prefer $B$ to $C$, and a majority prefer $C$ to $A$; the smallest margin of victory is for $A$ over $B$, so we drop this majority preference, yielding $BCA$.
  • Figure 2: RLCHF using aggregated rankings. The core addition to the standard RLHF process is the call-out of an explicit social welfare function, $F$, which determines how preferences are aggregated.
  • Figure 3: RLCHF using evaluator features and aggregated ranks. We show how an individuals' features can be used as an additional input to reward models within the RLHF process.
  • Figure 4: Supervised Learning from Simulated Collective Decisions. We show that with an individual or cardinal reward model, as presented in \ref{['fig:rlchf-f']}, responses $y$ to a prompt $x$ can be simulated. This process expands the scope of studying preferences within RLHF and opens future work on personalization and other topics.