BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment

TL;DR

This work identifies that the learned LLM should adhere to the proximity of the behavior LLM, which collects the training samples, and proposes online Preference Optimization in proximity to the Behavior LLM (BPO), emphasizing the importance of constructing a proper trust region for LLM alignment.

Abstract

Direct alignment from preferences (DAP) has emerged as a promising paradigm for aligning large language models (LLMs) to human desiderata from pre-collected, offline preference datasets. While recent studies indicate that existing offline DAP methods can directly benefit from online training samples, we highlight the need to develop specific online DAP algorithms to fully harness the power of online training. Specifically, we identify that the learned LLM should adhere to the proximity of the behavior LLM, which collects the training samples. To this end, we propose online Preference Optimization in proximity to the Behavior LLM (BPO), emphasizing the importance of constructing a proper trust region for LLM alignment. We conduct extensive experiments to validate the effectiveness and applicability of our approach by integrating it with various DAP methods, resulting in significant performance improvements across a wide range of tasks when training with the same amount of preference data. Even when only introducing one additional data collection phase, our online BPO improves its offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on Anthropic Helpfulness in terms of win rate against human reference text.

Figures (7)

• Figure 1: Given the same annotation budget, our $\mathcal{B}$PO (when $F=2$) significantly outperforms offline DPO ($F = 1$) on both TL;DR and Anthropic Helpfulness by introducing only one additional preference annotation phase. Its performance (when $F = 2$) even matches, if not exceeds, that of on-policy DPO ($F = T$), which collects new annotations at every step.
• Figure 2: Overview of the training pipeline of our $\mathcal{B}$PO. Our training loss $\mathcal{L}_\mathrm{\mathcal{B}PO}$ is calculated by constraining the KL divergence between $\pi_{\theta}$ and the behavior LLM $\pi_{\beta}$. Every $K$ step, we update $\pi_{\beta}$ with $\pi_{\theta}$ and use it to collect new samples for annotations.
• Figure 3: Aggregate metrics agarwal2021deep evaluating the win rate against human references with 95% confidence intervals (CIs), based on results reported in \ref{['tab:main-result']}. The CIs are estimated using the percentile bootstrap with stratified sampling. Higher median, IQM, and mean scores correspond to better performance. Our $\mathcal{B}$PO outperforms offline and on-policy DAP methods by a significant margin based on all metrics.
• Figure 4: We experiment with different data collection frequency $F$ for our $\mathcal{B}$PO (DPO) on TL;DR (Left) and Helpfulness (Right). The error bar denotes the one std of the win rates across 3 random seeds. Our $\mathcal{B}$PO is applicable to a small $F$. Even with $F = 2$, our $\mathcal{B}$PO (DPO) significantly outperforms offline DPO and at least matches the performance of on-policy DPO on both tasks.
• Figure 5: Ablation study on the reference model $\pi_\mathrm{ref}$. Even by setting $\pi_\mathrm{ref}$ as an optimized LLM $\pi_\mathrm{gold}$ that is significantly better than SFT LM $\pi_\mathrm{sft}$, on-policy DPO still under-performs our on-policy $\mathcal{B}$PO, validating that our improvement comes from constraining the divergence between the learned LLM $\pi_\theta$ and the behavior LLM $\pi_\beta$. The shaded area denotes one std.