One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF
Xin Cai
TL;DR
This work introduces Generalized Reinforce Optimization (GRO) as a unifying framework that bridges RL-based and RL-free approaches to RLHF and Large Reasoning Models. By reframing RLHF as neural bandit structured prediction and revisiting PPO under deterministic transitions, GRO provides a single objective that subsumes existing methods (e.g., RLOO, GRPO, ReMax, REINFORCE++; DPO, CPL, KTO) through flexible weighting and anchor-based separation. The framework aims to improve exploration, diversity, and sample efficiency while enabling seamless mixing of offline and online data. The authors invite empirical validation and feedback to assess GRO’s practical impact on RLHF and LRMs.
Abstract
In this article, we primarily examine a variety of RL-based and RL-free methods designed to address Reinforcement Learning from Human Feedback (RLHF) and Large Reasoning Models (LRMs). We begin with a concise overview of the typical steps involved in RLHF and LRMs. Next, we reinterpret several RL-based and RL-free algorithms through the perspective of neural structured bandit prediction, providing a clear conceptual framework that uncovers a deeper connection between these seemingly distinct approaches. Following this, we briefly review some core principles of reinforcement learning, drawing attention to an often-overlooked aspect in existing RLHF studies. This leads to a detailed derivation of the standard RLHF objective within a full RL context, demonstrating its equivalence to neural structured bandit prediction. Finally, by reinvestigating the principles behind Proximal Policy Optimization (PPO), we pinpoint areas needing adjustment, which culminates in the introduction of the Generalized Reinforce Optimization (GRO) framework, seamlessly integrating RL-based and RL-free methods in RLHF. We look forward to the community's efforts to empirically validate GRO and invite constructive feedback.