Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game

Barna Pásztor, Thomas Kleine Buening, Andreas Krause

TL;DR

This work reframes alignment as a sequential Stackelberg game between a Leader and a Follower to overcome the limitations of scalar reward modeling in RLHF, particularly with intransitive preferences. It introduces StackelbergGDA, a two-timescale gradient method that computes a unique Stackelberg equilibrium and scales to large language models, while enabling inference-time refinement through conditional sampling. Empirically, the Follower consistently improves outputs and transfers refinements across model families without extra fine-tuning, and the approach demonstrates strong performance on diverse preference datasets and general-purpose fine-tuning. The framework offers practical benefits for robust, consistent preference alignment and provides avenues for personalized test-time refinement, albeit with limitations around preference specification and convergence guarantees.

Abstract

We introduce Stackelberg Learning from Human Feedback (SLHF), a new framework for preference optimization. SLHF frames the alignment problem as a sequential-move game between two policies: a Leader, which commits to an action, and a Follower, which responds conditionally on the Leader's action. This approach decomposes preference optimization into a refinement problem for the Follower and an optimization problem against an adversary for the Leader. Unlike Reinforcement Learning from Human Feedback (RLHF), which assigns scalar rewards to actions, or Nash Learning from Human Feedback (NLHF), which seeks a simultaneous-move equilibrium, SLHF leverages the asymmetry of sequential play to capture richer preference structures. The sequential design of SLHF naturally enables inference-time refinement, as the Follower learns to improve the Leader's actions, and these refinements can be leveraged through iterative sampling. We compare the solution concepts of SLHF, RLHF, and NLHF, and lay out key advantages in consistency, data sensitivity, and robustness to intransitive preferences. Experiments on large language models demonstrate that SLHF achieves strong alignment across diverse preference datasets, scales from 0.5B to 8B parameters, and yields inference-time refinements that transfer across model families without further fine-tuning.

Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game

TL;DR

This work reframes alignment as a sequential Stackelberg game between a Leader and a Follower to overcome the limitations of scalar reward modeling in RLHF, particularly with intransitive preferences. It introduces StackelbergGDA, a two-timescale gradient method that computes a unique Stackelberg equilibrium and scales to large language models, while enabling inference-time refinement through conditional sampling. Empirically, the Follower consistently improves outputs and transfers refinements across model families without extra fine-tuning, and the approach demonstrates strong performance on diverse preference datasets and general-purpose fine-tuning. The framework offers practical benefits for robust, consistent preference alignment and provides avenues for personalized test-time refinement, albeit with limitations around preference specification and convergence guarantees.

Abstract

We introduce Stackelberg Learning from Human Feedback (SLHF), a new framework for preference optimization. SLHF frames the alignment problem as a sequential-move game between two policies: a Leader, which commits to an action, and a Follower, which responds conditionally on the Leader's action. This approach decomposes preference optimization into a refinement problem for the Follower and an optimization problem against an adversary for the Leader. Unlike Reinforcement Learning from Human Feedback (RLHF), which assigns scalar rewards to actions, or Nash Learning from Human Feedback (NLHF), which seeks a simultaneous-move equilibrium, SLHF leverages the asymmetry of sequential play to capture richer preference structures. The sequential design of SLHF naturally enables inference-time refinement, as the Follower learns to improve the Leader's actions, and these refinements can be leveraged through iterative sampling. We compare the solution concepts of SLHF, RLHF, and NLHF, and lay out key advantages in consistency, data sensitivity, and robustness to intransitive preferences. Experiments on large language models demonstrate that SLHF achieves strong alignment across diverse preference datasets, scales from 0.5B to 8B parameters, and yields inference-time refinements that transfer across model families without further fine-tuning.

Paper Structure

This paper contains 44 sections, 1 theorem, 21 equations, 3 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning from Human Feedback (RLHF).
  4. Limitations of Reward Modeling.
  5. Preference Optimization.
  6. Inference-Time Preference Improvement.
  7. Problem Statement
  8. Background on Existing Solution Concepts and Approaches
  9. Reinforcement Learning from Human Feedback (RLHF).
  10. Nash Learning from Human Feedback (NLHF).
  11. Stackelberg Learning from Human Feedback (SLHF)
  12. Comparison of Solution Concepts
  13. RLHF Solution.
  14. Nash Equilibrium.
  15. Stackelberg Equilibrium.
  16. ...and 29 more sections

Key Result

Proposition 1

Let $\tau^L,\tau^F > 0$ and suppose that $\pi^\textnormal{ref}(y \mid x) > 0$ for all $(x, y) \in {\mathcal{X}} \times {\mathcal{Y}}$. For any preference function $p(y \succ y' \mid x)$ there exists a unique solution $(\pi^\star, \omega^\star)$ to the preference optimization problem in eq:stackelber

Figures (3)

  • Figure 1: Transitive individual annotator preferences over three options $\{A, B, C\}$.
  • Figure 1: Prompt templates used to train a single-model for both Leader and Follower completions.
  • Figure 2: Directed graph based with completions generated by the fine-tuned models and edge directions representing the preference between them.

Theorems & Definitions (4)

  • Proposition 1
  • Remark 2
  • proof
  • proof