It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

TL;DR

This work identifies a saturation phenomenon in RLHF where improved reward and policy models no longer translate to better RLHF performance. It introduces the notion of seamlessness between PM and RM and develops SEAM, an automatic, data-centric estimator of this seamlessness with three variants (Contrast, GPT, Adv) and a length penalty. Through data selection and model augmentation experiments, SEAM demonstrates meaningful gains in RLHF performance (up to 4.5% with SEAM-filtered data and 4% with SEAM-guided augmentation) and mitigates saturation by focusing learning on samples where PM and RM disagree. The findings highlight the importance of PM RM interaction, offering practical, scalable techniques to improve RLHF in real-world settings and guiding future work toward online RLHF and absolute seamness measures.

Abstract

Reinforcement Learning from Human Feedback (RLHF) involves training policy models (PMs) and reward models (RMs) to align language models with human preferences. Instead of focusing solely on PMs and RMs independently, we propose to examine their interactions during fine-tuning, introducing the concept of seamlessness. Our study starts with observing the saturation phenomenon, where continual improvements in RM and PM do not translate into RLHF progress. Our analysis shows that RMs fail to assign proper scores to PM responses, resulting in a 35% mismatch rate with human preferences, highlighting a significant discrepancy between PM and RM. To measure seamlessness between PM and RM without human effort, we propose an automatic metric, SEAM. SEAM quantifies the discrepancies between PM and RM judgments induced by data samples. We validate the effectiveness of SEAM in data selection and model augmentation. Our experiments demonstrate that (1) using SEAM-filtered data for RL training improves RLHF performance by 4.5%, and (2) SEAM-guided model augmentation results in a 4% performance improvement over standard augmentation methods.

Figures (9)

• Figure 1: We introduce the concept of Seamlessness to measure the discrepancies between reward and policy models as supported by human evaluation. To automate measuring the Seamlessness, we propose SEAM, an automated method for estimating seamlessness between PM and RM. We validate its effectiveness through two experimental settings: data selection and augmentation.
• Figure 2: We examine the relation between the RLHF performance and the quality of PMs and RMs, measured by $\mathcal{P}_{in}$ and $\mathcal{A}_{in}$, respectively. We can see a "saturation phenomeno": the continual improvements of RM/PM do not translate into RLHF improvements.
• Figure 3: Cross-validation of PM and RM quality using different datasets(3 random seeds). The performance of RM and PM remains consistent across benchmarks. (e.g., on $\mathcal{D}{rl}$, the PM achieves 96% of its performance on $\mathcal{D}_p$.)
• Figure 4: Agreement between reward and human preference is evaluated by comparing two responses (A and B) from two different policy models. The blue points indicate agreement between the reward and human preferences, while the red points represent mismatches. However, the results show that the RM fails to assign a proper score to the generation from PM.
• Figure 5: Compared to the RLHF performance of the full dataset, filter low-SEAM data further improves RLHF (3 random seeds).

Theorems & Definitions (1)

• definition 1