Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

TL;DR

The paper addresses the limitation of evaluating reward models purely via fixed pairwise ranking by proposing MRMBench, a Multi-dimensional Reward Model Benchmark with six probing tasks that span harmlessness, helpfulness, correctness, coherence, complexity, and verbosity. It introduces inference-time probing to identify which preference dimensions a reward model relies on and to provide a confidence metric for reward predictions, all without additional training. Empirical results show MRMBench correlates strongly with downstream LLM alignment and reveal challenges in simultaneously capturing all dimensions, while inference-time probing enables interpretable diagnostics and improves alignment outcomes. The work suggests multi-objective optimization is needed in reward modeling and provides practical tools for more transparent and robust alignment of LLMs.

Abstract

Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with the alignment performance of large language models (LLMs), making it a reliable reference for developing advanced reward models. Our analysis of MRMBench evaluation results reveals that reward models often struggle to capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Additionally, our findings show that the proposed inference-time probing method offers a reliable metric for assessing the confidence of reward predictions, which ultimately improves the alignment of LLMs.

Figures (11)

• Figure 1: Sub-figure (a) illustrates the architecture of a reward model, in which both the Transformer decoder and the linear layer are typically trained using preference data. Sub-figure (b) depicts the process of probing preference representations. We design a classifier that takes the extracted preference representation as input and performs a probing task.
• Figure 2: The correlation between the aligned LLM win rate and the reward model's accuracy on MRMBench-Hard. Each point on the scatter plot represents a distinct reward model.
• Figure 3: Quantitative distance distributions to the centroids of each preference dimension for several input-response pairs. A dark color means a smaller distance from the centroid, as computed in Eq. \ref{['eq:distance-score']} in the distribution. Further results for additional input-response pairs can be found in Figure \ref{['fig:heatmap_appendix_with_case']}.
• Figure 4: Sub-figure (a) illustrates the evaluation rewards (denoted as EvalReward) for aligning the LLaMA-3.1-8B-SFT using different reward methods. We report the average results along with their standard deviation. Sub-figure (b) shows the performance of aligned LLMs on the test set for one of the seeds. ITP: Inference-time probing.
• Figure 5: The correlation between the human-labeled win rate and the reward model’s accuracy on MRMBench-Hard. Each point on the scatter plot represents a distinct reward model.