Solving the Inverse Alignment Problem for Efficient RLHF
Shambhavi Krishna, Aishwarya Sahoo
TL;DR
The paper defines the inverse alignment problem for RLHF, arguing that static reward models trained on pooled offline preferences can provide weak signals as policy distributions shift during training. It introduces Filtered Reward Fine-Tuning (FRFT), a framework that pauses RLHF to filter offline preference data using Sentence-BERT embeddings aligned with the current policy and then fine-tunes the reward model via a Bradley–Terry objective; FRFT can be iterated as FRFT-$\alpha$. Empirically, FRFT improves alignment and accelerates convergence compared to vanilla RLHF, achieving competitive performance with as few as 2000 filtered records versus 75000 random records, and enabling on-policy reward feedback with reduced compute. The approach highlights the practical potential of on-policy feedback loops in RLHF and points to future work on more powerful embeddings and end-to-end reward-model optimization.
Abstract
Collecting high-quality preference datasets for reinforcement learning from human feedback (RLHF) is resource-intensive and challenging. As a result, researchers often train reward models on extensive offline datasets which aggregate diverse generation sources and scoring/alignment policies. We hypothesize that this aggregation has an averaging effect on reward model scores, which limits signal and impairs the alignment process. Inspired by the field of inverse RL, we define the 'inverse alignment problem' in language model training, where our objective is to optimize the critic's reward for a fixed actor and a fixed offline preference dataset. We hypothesize that solving the inverse alignment problem will improve reward model quality by providing clearer feedback on the policy's current behavior. To that end, we investigate whether repeatedly fine-tuning a reward model on subsets of the offline preference dataset aligned with a periodically frozen policy during RLHF improves upon vanilla RLHF. Our empirical results demonstrate that this approach facilitates superior alignment and faster convergence compared to using an unaligned or out-of-distribution reward model relative to the LLM policy.