Automatically Finding Reward Model Biases

TL;DR

This work offers a simple approach of using an LLM to iteratively propose and refine candidate biases and shows evidence that evolutionary iteration outperforms flat best-of-N search and the recall of the pipeline using synthetically injected biases.

Abstract

Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.

Figures (10)

• Figure 1: Between the original response with normal spacing, and the alternative response with extra whitespace characters between words, the RM mistakenly prefers the latter, disagreeing with the LLM judge.
• Figure 2: Illustration of our pipeline. Each circle represents a population of candidate biases at different stages.
• Figure 3: Format biases of Skywork-V2-8B on two user prompt datasets. The numbers below are the mean reward difference (i.e. bias strength) and the 95% CI, pooled across all three rewriters. The red bar in the boxplot is the mean.
• Figure 4: Comparison of visualization and DABS metrics.
• Figure 5: Recall rates under different conditions. The confidence intervals are 95% Wilson CIs with $n=10$, hence the wide error bars.