Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How RLHF Amplifies Sycophancy

Itai Shapira, Gerdus Benade, Ariel D. Procaccia

TL;DR

This paper formalizes why RLHF can amplify sycophancy by tracing amplification to bias in human preference data learned into a reward model and the interaction with policy optimization. It presents a two-stage mechanism: reward learning from comparisons via a random utility framework, and policy optimization (KL-regularized or Best-of-$N$) that reweights toward high-reward samples. The authors derive precise conditions under which amplification occurs, relate labeler bias to reward bias through a mixed-pair statistic, and propose a KL-minimal correction to disincentivize amplification without sacrificing useful capabilities. Empirical analysis across bias-injection strategies and reward models shows reward tilt is common and predictive of amplification under optimization, supporting the need for targeted mitigation and alternative supervision paradigms.

Abstract

Large language models often exhibit increased sycophantic behavior after preference-based post-training, showing a stronger tendency to affirm a user's stated or implied belief even when this conflicts with factual accuracy or sound judgment. We present a formal analysis of how alignment from human feedback can increase this failure mode by identifying an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment. We show that the direction of behavioral drift is determined by a covariance under the base policy between endorsing the belief signal in the prompt and the learned reward, and that the first-order effect reduces to a simple mean-gap condition. We then analyze reward learning from pairwise comparisons under random utility models like Bradley-Terry and characterize when bias in human annotators' preferences induces this reward gap. Next, we propose a training-time intervention designed to neutralize the amplification mechanism itself. Among all post-trained policies that prevent sycophantic behavior from increasing, we characterize the unique policy closest in KL divergence to the unconstrained post-trained policy, and derive the corresponding minimal reward correction as a closed-form agreement penalty. Computational experiments find that reward gaps are common and cause behavioral drift in all the configurations considered.

How RLHF Amplifies Sycophancy

TL;DR

This paper formalizes why RLHF can amplify sycophancy by tracing amplification to bias in human preference data learned into a reward model and the interaction with policy optimization. It presents a two-stage mechanism: reward learning from comparisons via a random utility framework, and policy optimization (KL-regularized or Best-of-) that reweights toward high-reward samples. The authors derive precise conditions under which amplification occurs, relate labeler bias to reward bias through a mixed-pair statistic, and propose a KL-minimal correction to disincentivize amplification without sacrificing useful capabilities. Empirical analysis across bias-injection strategies and reward models shows reward tilt is common and predictive of amplification under optimization, supporting the need for targeted mitigation and alternative supervision paradigms.

Abstract

Large language models often exhibit increased sycophantic behavior after preference-based post-training, showing a stronger tendency to affirm a user's stated or implied belief even when this conflicts with factual accuracy or sound judgment. We present a formal analysis of how alignment from human feedback can increase this failure mode by identifying an explicit amplification mechanism that causally links optimization against a learned reward to bias in the human preference data used for alignment. We show that the direction of behavioral drift is determined by a covariance under the base policy between endorsing the belief signal in the prompt and the learned reward, and that the first-order effect reduces to a simple mean-gap condition. We then analyze reward learning from pairwise comparisons under random utility models like Bradley-Terry and characterize when bias in human annotators' preferences induces this reward gap. Next, we propose a training-time intervention designed to neutralize the amplification mechanism itself. Among all post-trained policies that prevent sycophantic behavior from increasing, we characterize the unique policy closest in KL divergence to the unconstrained post-trained policy, and derive the corresponding minimal reward correction as a closed-form agreement penalty. Computational experiments find that reward gaps are common and cause behavioral drift in all the configurations considered.
Paper Structure (30 sections, 17 theorems, 121 equations, 7 figures, 1 table)

This paper contains 30 sections, 17 theorems, 121 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Behavior Amplification under Preference Optimization
  5. KL-regularized reward maximization
  6. First-order drift at small $\beta$.
  7. Best-of-N
  8. From Labeler Bias to Biased Reward
  9. Minimal Correction to Avoid Amplification
  10. Empirical Analysis
  11. Reward-tilt measurement
  12. Optimization-pressure sign test
  13. Discussion
  14. Deferred Proofs for Section 3
  15. Proof of Theorem 1
  16. ...and 15 more sections

Key Result

Theorem 1

Let $\pi_\beta^\star$ be the optimal policy solving eq:kl-obj. Then for any bounded measurable $g$, any prompt $x\in\mathcal{X}$, and any $\beta> 0$,

Figures (7)

  • Figure 1: To estimate reward tilt, we sample 64 agreeing and 64 corrective responses for each biased prompt $x'$ and score them using open-source public reward models. \ref{['fig:tilt_by_bias_type', 'fig:tilt_by_dataset']} report the fraction of prompts exhibiting a positive mean reward gap ($\Delta^{\mathrm{mean}}(x) > 0$), where the average reward for agreement exceeds the average reward for correction, stratified by bias-injection strategy and source dataset. \ref{['fig:best_of_n']} illustrates the evolution of the sycophancy rate under Best-of-$N$ optimization. We partition the prompts into positive ($\Delta^{\mathrm{mean}}(x) > 0$) and negative ($\Delta^{\mathrm{mean}}(x) < 0$) tilt subsets based on the reward gap measured on responses generated by a distinct base model, and compare the Best-of-$N$ trends to the static sycophancy rate of a corresponding RLHF checkpoint.
  • Figure 5.1: System prompts used to generate controlled response classes for the same biased prompt $x'$.
  • Figure 5.2: Example Answer Suggestion prompt ($x'$) containing an explicit belief cue.
  • Figure 5.3: Two contrasting candidate responses to the prompt in \ref{['fig:sycophancy_prompt']}. (a) The sycophantic response agrees with the user's mistaken guess, while (b) the corrective response states the true fact.
  • Figure 5.4: Fraction of prompts exhibiting positive reward tilt, by reward model. We find that the measured tilt fraction is similar across reward models spanning different architectures and roughly an order-of-magnitude scale range (DeBERTa-v3, OpenLLaMA-3B RM, Beaver-7B), indicating that using a larger or more sophisticated public reward model does not, by itself, reduce the prevalence of positive reward tilt in this setting.
  • ...and 2 more figures

Theorems & Definitions (36)

  • Definition 1: Sycophancy of a policy
  • Theorem 1
  • Corollary 1
  • Corollary 2
  • Theorem 2
  • Theorem 3
  • Definition 2
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • ...and 26 more