Reward Models Identify Consistency, Not Causality

TL;DR

The paper examines whether state-of-the-art reward models truly evaluate causal reasoning or merely assess coherence in reasoning trajectories. Through systematic input perturbations (question removal, shuffling, and step truncation) across multiple datasets, the authors show that rewards reflect internal coherence and completeness of reasoning rather than explicit problem comprehension, with complete trajectories driving scores. They demonstrate a consistency bias in RM outputs and rankings, suggesting limited generalization to novel problem distributions. The work argues for causality-aware reward modeling, proposing avenues such as counterfactual training, chain-of-thought awareness, and human-in-the-loop safeguards to improve true logical validity assessment in evaluations.

Abstract

Reward models (RMs) play a crucial role in aligning large language models (LLMs) with human preferences and enhancing reasoning quality. Traditionally, RMs are trained to rank candidate outputs based on their correctness and coherence. However, in this work, we present several surprising findings that challenge common assumptions about RM behavior. Our analysis reveals that state-of-the-art reward models prioritize structural consistency over causal correctness. Specifically, removing the problem statement has minimal impact on reward scores, whereas altering numerical values or disrupting the reasoning flow significantly affects RM outputs. Furthermore, RMs exhibit a strong dependence on complete reasoning trajectories truncated or incomplete steps lead to significant variations in reward assignments, indicating that RMs primarily rely on learned reasoning patterns rather than explicit problem comprehension. These findings hold across multiple architectures, datasets, and tasks, leading to three key insights: (1) RMs primarily assess coherence rather than true reasoning quality; (2) The role of explicit problem comprehension in reward assignment is overstated; (3) Current RMs may be more effective at ranking responses than verifying logical validity. Our results suggest a fundamental limitation in existing reward modeling approaches, emphasizing the need for a shift toward causality-aware reward models that go beyond consistency-driven evaluation.

Figures (4)

• Figure 1: Illustrations of the reward model input modifications. We apply various perturbations, including numerical value modification, question shuffling, question truncation, initial steps truncation, last steps truncation, and all steps truncation, to assess the sensitivity of different input components on reward evaluation.
• Figure 2: Density maps of absolute reward errors across different datasets when truncating various input components, using Qwen-2.5-Math-1.5B-Instruct as the base model and Skywork-O1-Open-PRM-Qwen-2.5-1.5B as the reward model..
• Figure 3: Density maps of absolute reward errors across different datasets. We compare the effects of question truncation, question shuffling, and numerical value modification using Qwen-2.5-Math-1.5B-Instruct as the base model and Skywork-O1-Open-PRM-Qwen-2.5-1.5B as the reward model.
• Figure 4: Spearman correlation of reward rankings under input truncation and Shuffling on MATH500 and Olympiad Bench. $N$ is the number of sampled trajectories for each input question.