Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration

TL;DR

This paper tackles sample-efficient alignment of language models via reinforcement learning from human feedback (RLHF) by introducing Hybrid Preference Optimization (HPO). HPO blends online exploration with offline preference data, relaxing the strict concentrability constraints of purely offline methods while leveraging offline data to accelerate online learning. The authors prove provable upper bounds on the KL-regularized objective gap, establish lower bounds for pure offline and online RLHF, and demonstrate that HPO achieves improved sample efficiency, especially in linear MDP settings. Empirical results in a linear contextual bandit setup corroborate the theoretical gains, showing reduced online sample needs when offline data is available, which has practical implications for scalable, cost-effective RLHF deployment.

Abstract

Reinforcement Learning from Human Feedback (RLHF) is currently the leading approach for aligning large language models with human preferences. Typically, these models rely on extensive offline preference datasets for training. However, offline algorithms impose strict concentrability requirements, which are often difficult to satisfy. On the other hand, while online algorithms can avoid the concentrability issue, pure online exploration could be expensive due to the active preference query cost and real-time implementation overhead. In this paper, we propose a novel approach: Hybrid Preference Optimization (HPO) which combines online exploration with existing offline preferences by relaxing the stringent concentrability conditions for offline exploration, as well as significantly improving the sample efficiency for its online counterpart. We give the first provably optimal theoretical bound for Hybrid RLHF with preference feedback, providing sample complexity bounds for policy optimization with matching lower bounds. Our results yield improved sample efficiency of hybrid RLHF over pure offline and online exploration.

Figures (2)

• Figure 1: (Left) We plot the cumulative regret as the number of online samples grows. (Right) We plot the suboptimality gaps for all online, offline and hybrid algorithms as a function of total number of samples. Since HPO has access to offline samples, to ensure fairness while comparinng, for the hybrid case we report performance with number of offline + online samples seen. HPO used an offline dataset of size $N_{\rm off} = 500$, hence in both the plots the value corresponding to 750, is after 250 additional online samples seen, the value corresponding to 1000, is after 500 additional online samples collected by HPO and so on.
• Figure 2: Simulating linDB feedback with linB feedback

Theorems & Definitions (18)

• Definition 4.1
• Definition 4.2
• Remark
• Theorem 1
• Definition 5.1
• Theorem 2: Sample Complexity Lower Bound for Offline RLHF
• Theorem 3: Sample Complexity Lower Bound for Online RLHF
• Theorem 4
• proof
• Lemma 1: Concentration for Algorithm \ref{['alg:hybrid_rlhf']}