Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On a Connection Between Imitation Learning and RLHF

Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, Vasant G Honavar

TL;DR

By revealing that RLHF effectively performs imitation learning on the chosen-response distribution under a reverse KL objective, the paper unifies RLHF and imitation learning. It then introduces Direct Imitation Learning (DIL), a principled framework that directly optimizes a density-ratio reward without explicit reward modeling or RL loops, leveraging Bregman-divergence–based density ratio estimation. DIL subsumes existing alignment methods like DPO and demonstrates strong empirical gains on Open LLM Leaderboard benchmarks and human-preference–driven tasks (Reddit TL;DR, Anthropic-HH). The results suggest that avoiding BT assumptions and directly optimizing the imitation objective yields better preservation of reasoning abilities and closer alignment with human preferences.

Abstract

This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution. Building on this connection, we propose DIL, a principled framework that directly optimizes the imitation learning objective. DIL provides a unified imitation learning perspective on alignment, encompassing existing alignment algorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods on various challenging benchmarks.

On a Connection Between Imitation Learning and RLHF

TL;DR

By revealing that RLHF effectively performs imitation learning on the chosen-response distribution under a reverse KL objective, the paper unifies RLHF and imitation learning. It then introduces Direct Imitation Learning (DIL), a principled framework that directly optimizes a density-ratio reward without explicit reward modeling or RL loops, leveraging Bregman-divergence–based density ratio estimation. DIL subsumes existing alignment methods like DPO and demonstrates strong empirical gains on Open LLM Leaderboard benchmarks and human-preference–driven tasks (Reddit TL;DR, Anthropic-HH). The results suggest that avoiding BT assumptions and directly optimizing the imitation objective yields better preservation of reasoning abilities and closer alignment with human preferences.

Abstract

This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution. Building on this connection, we propose DIL, a principled framework that directly optimizes the imitation learning objective. DIL provides a unified imitation learning perspective on alignment, encompassing existing alignment algorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods on various challenging benchmarks.

Paper Structure

This paper contains 24 sections, 1 theorem, 31 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notations and Preliminaries
  4. Methodology
  5. RLHF is a Form of Imitation Learning
  6. Direct Imitation Learning
  7. Density Ratio Reward Estimation
  8. Optimization
  9. Discussion: DPO is a special case of DIL
  10. Experiments
  11. Experimental Results
  12. Performance Comparison on Benchmarks
  13. Performance Comparison with Human Preferences
  14. Further Analysis
  15. Conclusion
  16. ...and 9 more sections

Key Result

Proposition 4.1

Suppose the chosen response distribution $p(\mathbf{y}\mid \mathbf{x})$, the EBM $\pi_{\boldsymbol{\phi}}(\mathbf{y}\mid \mathbf{x})$, and the model $\pi_{\boldsymbol{\theta}}(\mathbf{y}\mid \mathbf{x})$. KL-regularized RLHF with $\beta=1$ can be viewed as the following problem: where $\pi_{\rm{chosen}}(\mathbf{y}\mid \mathbf{x}) = \pi_{\boldsymbol{\phi}}(\mathbf{y}\mid \mathbf{x}) = \pi_{\boldsy

Figures (2)

  • Figure 1: The illustration of different losses (LSIF, BCE, and UKL), as shown in Table \ref{['tab:h_divergences']}.
  • Figure 2: The training dynamics of DIL variants, DPO and SimPO on Mistral show that DIL exhibits the smallest decline in chosen likelihoods, while still increasing the likelihood margins between rejected and chosen responses, compared to SimPO and DPO. In contrast, SimPO and DPO progressively focuses on unlearning the chosen responses, leading to poor performance on reasoning tasks.

Theorems & Definitions (1)

  • Proposition 4.1