Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Axioms for AI Alignment from Human Feedback

Luise Ge, Daniel Halpern, Evi Micha, Ariel D. Procaccia, Itai Shapira, Yevgeniy Vorobeychik, Junlin Wu

TL;DR

It is demonstrated that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms, and novel rules for learning reward functions with strong axiomatic guarantees are developed.

Abstract

In the context of reinforcement learning from human feedback (RLHF), the reward function is generally derived from maximum likelihood estimation of a random utility model based on pairwise comparisons made by humans. The problem of learning a reward function is one of preference aggregation that, we argue, largely falls within the scope of social choice theory. From this perspective, we can evaluate different aggregation methods via established axioms, examining whether these methods meet or fail well-known standards. We demonstrate that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms. In response, we develop novel rules for learning reward functions with strong axiomatic guarantees. A key innovation from the standpoint of social choice is that our problem has a linear structure, which greatly restricts the space of feasible rules and leads to a new paradigm that we call linear social choice.

Axioms for AI Alignment from Human Feedback

TL;DR

It is demonstrated that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms, and novel rules for learning reward functions with strong axiomatic guarantees are developed.

Abstract

In the context of reinforcement learning from human feedback (RLHF), the reward function is generally derived from maximum likelihood estimation of a random utility model based on pairwise comparisons made by humans. The problem of learning a reward function is one of preference aggregation that, we argue, largely falls within the scope of social choice theory. From this perspective, we can evaluate different aggregation methods via established axioms, examining whether these methods meet or fail well-known standards. We demonstrate that both the Bradley-Terry-Luce Model and its broad generalizations fail to meet basic axioms. In response, we develop novel rules for learning reward functions with strong axiomatic guarantees. A key innovation from the standpoint of social choice is that our problem has a linear structure, which greatly restricts the space of feasible rules and leads to a new paradigm that we call linear social choice.
Paper Structure (22 sections, 13 theorems, 39 equations, 1 figure, 1 table)

This paper contains 22 sections, 13 theorems, 39 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Our Approach
  3. Our Results
  4. Related Work
  5. The Linear Social Choice Model
  6. Loss-Based Rules
  7. Standard Loss Formulation
  8. Majority-Based Loss Formulation
  9. Social Choice-Based Rule
  10. Discussion
  11. Deferred Proofs
  12. Proof of \ref{['lem:unconstr']}
  13. Proof of \ref{['lem:core']}
  14. Proof of \ref{['lem:opt0']}
  15. Proof of \ref{['lem:continuity']}
  16. ...and 7 more sections

Key Result

Theorem 3.1

If a linear rank aggregation rule $f$ optimizes a loss function $\ell$ that satisfies $\inf_x \ell(x) < \ell(0)$ and is either nondecreasing and weakly convex, or strictly convex (and possibly nonmonotone), then $f$ fails PMC and PO.

Figures (1)

  • Figure 1: Graph showing pairwise majority relationship between candidates. Regular edges show relationships among $c_i^+$ candidates and among $c_i^-$ candidates. Thick edges indicate that $c^*$ pairwise beats all candidates, and each $c_i^+$ pairwise beats each $c_j^-$ candidate.

Theorems & Definitions (31)

  • Definition 2.1: Pareto Optimality
  • Definition 2.2: Pairwise Majority Consistency (PMC)
  • Theorem 3.1
  • proof
  • Lemma 3.2
  • Lemma 3.3
  • Lemma 3.4
  • Lemma 3.5
  • Theorem 3.6
  • Theorem 3.7
  • ...and 21 more