Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reward Models are Metrics in a Trench Coat

Sebastian Gehrmann

TL;DR

This paper argues that reward models and evaluation metrics for LLM outputs have evolved largely in isolation, despite pursuing similar goals of aligning model behavior with human preferences. It combines citation analysis with two targeted experiments to show that cross-field evaluation can yield valuable insights, with metrics sometimes outperforming reward-model baselines and vice versa. The authors provide a comprehensive survey of both fields, identify opportunities in preference elicitation, avoidance of spurious correlations, and meta-evaluation, and offer concrete recommendations to foster collaboration without collapsing into a monoculture. Emphasizing sociotechnical context, calibration, and robust benchmarking, the work outlines practical paths for integrating advances across fields to improve safety, reliability, and usefulness of post-training language-model systems.

Abstract

The emergence of reinforcement learning in post-training of large language models has sparked significant interest in reward models. Reward models assess the quality of sampled model outputs to generate training signals. This task is also performed by evaluation metrics that monitor the performance of an AI model. We find that the two research areas are mostly separate, leading to redundant terminology and repeated pitfalls. Common challenges include susceptibility to spurious correlations, impact on downstream reward hacking, methods to improve data quality, and approaches to meta-evaluation. Our position paper argues that a closer collaboration between the fields can help overcome these issues. To that end, we show how metrics outperform reward models on specific tasks and provide an extensive survey of the two areas. Grounded in this survey, we point to multiple research topics in which closer alignment can improve reward models and metrics in areas such as preference elicitation methods, avoidance of spurious correlations and reward hacking, and calibration-aware meta-evaluation.

Reward Models are Metrics in a Trench Coat

TL;DR

This paper argues that reward models and evaluation metrics for LLM outputs have evolved largely in isolation, despite pursuing similar goals of aligning model behavior with human preferences. It combines citation analysis with two targeted experiments to show that cross-field evaluation can yield valuable insights, with metrics sometimes outperforming reward-model baselines and vice versa. The authors provide a comprehensive survey of both fields, identify opportunities in preference elicitation, avoidance of spurious correlations, and meta-evaluation, and offer concrete recommendations to foster collaboration without collapsing into a monoculture. Emphasizing sociotechnical context, calibration, and robust benchmarking, the work outlines practical paths for integrating advances across fields to improve safety, reliability, and usefulness of post-training language-model systems.

Abstract

The emergence of reinforcement learning in post-training of large language models has sparked significant interest in reward models. Reward models assess the quality of sampled model outputs to generate training signals. This task is also performed by evaluation metrics that monitor the performance of an AI model. We find that the two research areas are mostly separate, leading to redundant terminology and repeated pitfalls. Common challenges include susceptibility to spurious correlations, impact on downstream reward hacking, methods to improve data quality, and approaches to meta-evaluation. Our position paper argues that a closer collaboration between the fields can help overcome these issues. To that end, we show how metrics outperform reward models on specific tasks and provide an extensive survey of the two areas. Grounded in this survey, we point to multiple research topics in which closer alignment can improve reward models and metrics in areas such as preference elicitation methods, avoidance of spurious correlations and reward hacking, and calibration-aware meta-evaluation.

Paper Structure

This paper contains 26 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. How did modern Reward Models come about?
  3. Quantifying the Research Field Separation
  4. What can one learn from the other?
  5. Metrics can perform well on Reward Model Benchmarks
  6. Reward Models can underperform on Metrics Benchmarks
  7. Metrics and reward models are (not) the same
  8. Designing Reward Models and Evaluation Metrics
  9. Sociotechnical Context matters
  10. Aspect-aligned reward models
  11. Training Reward Models and Evaluation Metrics
  12. Data Collection
  13. Optimization targets
  14. Testing Reward Models and Evaluation Metrics
  15. Identifying and Debugging Reward Hacking
  16. ...and 11 more sections

Figures (3)

  • Figure 1: The figure shows the number of publications per year in the three subfields according to a keyword search on Google Scholar. Publications on evaluation metrics have slowed, even though research on reward modeling and LLM-as-a-judge is quickly rising in popularity. If the fields were actively learning from one another, one could assume that mentions of "evaluation metrics" should be growing alongside these newly emerging fields, but they are not.
  • Figure 2: In our analysis of citation dynamics across the three fields, we find that evaluation papers tend to cite other evaluation papers across research fields, while reward model papers mostly cite each other and are highly focused on machine learning venues. LLM-as-a-judge work mostly cites ML and NLP venues, but has less clear citation dynamics.
  • Figure 3: We show the percentage of citations to papers that were published more than three years ago. Reward model literature exhibits outlier behavior in which this percentage is decreasing drastically every year.