A Systematic Analysis of Base Model Choice for Reward Modeling

TL;DR

The paper tackles how base-model choice affects reward modeling in RLHF, showing that defaulting to $Llama-3.x$ can miss substantial performance gains. It systematically evaluates 40 base models using two RM formulations—Bradley-Terry with binary preferences and Regression with multi-attribute scores—and uses RewardBench as the evaluation benchmark, while also analyzing training stages and pre-training data distributions. Key findings include up to $14\%$ gains from base-model changes, an average of $+18\%$ in top-$5$ to top-$10$ model selection when combining a small set of benchmarks, a $+15.5\%$ gain from post-training (SFT) with certain alignment steps causing a $3$–$5\%$ drop, and a $+1.5\%$ reduction in regression error when incorporating estimated pre-training data distributions. These results offer practical guidance for RM practitioners on base-model selection, benchmark-based model prediction, and the nuanced effects of training stages and data distributions on reward modeling performance.

Abstract

Reinforcement learning from human feedback (RLHF) and, at its core, reward modeling have become a crucial part of training powerful large language models (LLMs). One commonly overlooked factor in training high-quality reward models (RMs) is the effect of the base model, which is becoming more challenging to choose given the rapidly growing pool of LLMs. In this work, we present a systematic analysis of the effect of base model selection on reward modeling performance. Our results show that the performance can be improved by up to 14% compared to the most common (i.e., default) choice. Moreover, we showcase the strong statistical relation between some existing benchmarks and downstream performances. We also demonstrate that the results from a small set of benchmarks could be combined to boost the model selection ($+$18% on average in the top 5-10). Lastly, we illustrate the impact of different post-training steps on the final performance and explore using estimated data distributions to reduce performance prediction error.

Figures (13)

• Figure 1: Ratio of the base models used in the top 30 entries of RewardBench (Dec 2024). Almost all the entries are trained on top of a small set of base models (e.g., Llama-3.x models comprise 50% of the entries).
• Figure 2: Reward Modeling Performance Gains. Relative gains are illustrated concerning the Llama-3.x model (marked as red) within the same group.
• Figure 3: Statistical Correlation w.r.t. Reward Modeling Performance. The subset benchmarks of Open LLM Leaderboard v2 (v1) are denoted with an $\ddagger$ ($\dagger$). Text Colors:Red$\rightarrow$ Aggregate benchmark, Green$\rightarrow$ Training metric.
• Figure 4: Benchmark's Coverage. We only retain benchmarks with at least 0.4 and 0.7 coverage at $k=5$ and $k=10$, respectively.
• Figure 5: Coefficients. Only five benchmarks are assigned a non-zero weight by the trained model. The topics of these benchmarks are as follows: Coding$\rightarrow$ MBPP+ and HumanEval+, Safety$\rightarrow$ ToxiGen, General$\rightarrow$ IFEval, and Training Metrics$\rightarrow$ #Params (see \ref{['app:data']} for more details).