Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
Chenliang Li, Siliang Zeng, Zeyi Liao, Jiaxiang Li, Dongyeop Kang, Alfredo Garcia, Mingyi Hong
TL;DR
This work tackles the alignment problem by proposing Alignment with Integrated Human Feedback (AIHF), a single-stage framework that jointly learns rewards and policies from both demonstrations and preferences. By formulating AIHF as a bi-level optimization, the authors unify supervised demonstration data with human preferences, providing a finite-time convergence guarantee and showing that the resulting reward and policy are consistent with all data sources. Special cases of AIHF recover RLHF, DPO, and self-play variants, while experiments on LLM alignment and MuJoCo robotics demonstrate substantial performance gains, especially when preference data are limited or demonstrations are abundant but imperfect. The approach reduces distribution mismatch and data under-utilization inherent in multi-stage pipelines, offering a principled, data-efficient path toward better-aligned AI systems with practical impact for both language models and embodied agents.
Abstract
Aligning human preference and value is an important requirement for building contemporary foundation models and embodied AI. However, popular approaches such as reinforcement learning with human feedback (RLHF) break down the task into successive stages, such as supervised fine-tuning (SFT), reward modeling (RM), and reinforcement learning (RL), each performing one specific learning task. Such a sequential approach results in serious issues such as significant under-utilization of data and distribution mismatch between the learned reward model and generated policy, which eventually lead to poor alignment performance. We develop a single stage approach named Alignment with Integrated Human Feedback (AIHF), capable of integrating both human preference and demonstration to train reward models and the policy. The proposed approach admits a suite of efficient algorithms, which can easily reduce to, and leverage, popular alignment algorithms such as RLHF and Directly Policy Optimization (DPO), and only requires minor changes to the existing alignment pipelines. We demonstrate the efficiency of the proposed solutions with extensive experiments involving alignment problems in LLMs and robotic control problems in MuJoCo. We observe that the proposed solutions outperform the existing alignment algorithms such as RLHF and DPO by large margins, especially when the amount of high-quality preference data is relatively limited.