How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective

TL;DR

This work tackles the problem of aligning large language models using offline human demonstrations without costly preference labeling or adversarial training. It introduces generalized self-imitation learning (GSIL), reframing imitation learning as a density-ratio estimation task and deriving a self-normalized, closed-form policy update that eliminates RL loops. GSIL supports a family of density-ratio losses and combines real demonstration data with self-generated data to drive learning, achieving significant improvements over SFT and SPIN, and even surpassing some DPO results on math, coding, and reasoning benchmarks, while also enhancing safety alignment. The approach offers a practical, scalable path for demonstration-based alignment and provides a unified perspective bridging imitation learning and density-ratio estimation for offline LLM fine-tuning.

Abstract

This paper introduces a novel generalized self-imitation learning ($\textbf{GSIL}$) framework, which effectively and efficiently aligns large language models with offline demonstration data. We develop $\textbf{GSIL}$ by deriving a surrogate objective of imitation learning with density ratio estimates, facilitating the use of self-generated data and optimizing the imitation learning objective with simple classification losses. $\textbf{GSIL}$ eliminates the need for complex adversarial training in standard imitation learning, achieving lightweight and efficient fine-tuning for large language models. In addition, $\textbf{GSIL}$ encompasses a family of offline losses parameterized by a general class of convex functions for density ratio estimation and enables a unified view for alignment with demonstration data. Extensive experiments show that $\textbf{GSIL}$ consistently and significantly outperforms baselines in many challenging benchmarks, such as coding (HuamnEval), mathematical reasoning (GSM8K) and instruction-following benchmark (MT-Bench).

Figures (10)

• Figure 1: Illustration of different characteristics of KL divergence. SFT exhibits mass-covering behavior by minimizing forward KL, while our GSIL exhibits mode-seeking behavior by minimizing reverse KL.
• Figure 2: The reward dynamics for SPIN and GSIL w/ Logistic on UltraFeedback show increasing margins between the rewards of real demonstrations and self-generated data. In SPIN, however, the rewards for real data drop below zero, while in GSIL, they continue to increase and stay positive. Results for other losses in the GSIL framework are provided in Figure \ref{['fig:app_rewards']} in Section \ref{['sec:exp']}.
• Figure 3: Results on MT-Bench with regard to different types of questions. We can observe GSIL shows significant gains in reasoning, math, and coding tasks and different trade-offs are imposed by different losses.
• Figure 4: The win rates, computed by GPT-4, in comparison to the chosen responses for Anthropic-HH one-step dialogue. Here, we utilize logistic loss for our GSIL as we observe similar performance across different losses.
• Figure 5: The training dynamics of real rewards of demonstration data and margins show that, for all our objectives, the margins between the rewards of real demonstrations and self-generated data keep increasing. Additionally, the rewards of real data continue to increase and remain positive.