PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling

TL;DR

PaTaRM tackles the mismatch between pairwise and pointwise reward models in RLHF by introducing a Preference-Aware Reward (PAR) mechanism and dynamic rubric adaptation. This unified framework converts relative pairwise preferences into robust pointwise supervision via judgment rollouts evaluated under adaptable rubrics, enabling end-to-end training without explicit pointwise labels. Empirical results show consistent improvements on RewardBench and RMBench across multiple Qwen model sizes, and substantial downstream RLHF gains on IFEval and InFoBench, especially for smaller models. The work advances interpretable, generalizable reward modeling with reduced annotation costs, enhancing the practicality of RLHF in open-ended tasks.

Abstract

Reward models (RMs) are central to reinforcement learning from human feedback (RLHF), providing the critical supervision signals that align large language models (LLMs) with human preferences. While generative reward models (GRMs) offer greater interpretability than traditional scalar RMs, current training paradigms remain limited. Pair-wise methods rely on binary good-versus-bad labels, which cause mismatches for point-wise inference and necessitate complex pairing strategies for effective application in RLHF. On the other hand, point-wise methods require more elaborate absolute labeling with rubric-driven criteria, resulting in poor adaptability and high annotation costs. In this work, we propose the Preference-Aware Task-Adaptive Reward Model (PaTaRM), a unified framework that integrates a preference-aware reward (PAR) mechanism with dynamic rubric adaptation. PaTaRM leverages relative preference information from pairwise data to construct robust point-wise training signals, eliminating the need for explicit point-wise labels. Simultaneously, it employs a task-adaptive rubric system that flexibly generates evaluation criteria for both global task consistency and instance-specific fine-grained reasoning. This design enables efficient, generalizable, and interpretable reward modeling for RLHF. Extensive experiments show that PaTaRM achieves an average relative improvement of 4.7% on RewardBench and RMBench across Qwen3-8B and Qwen3-14B models. Furthermore, PaTaRM boosts downstream RLHF performance, with an average improvement of 13.6% across IFEval and InFoBench benchmarks, confirming its effectiveness and robustness. Our code is available at https://github.com/JaneEyre0530/PaTaRM.

Figures (10)

• Figure 1: Challenges in two GRM Paradigms.
• Figure 2: Overview of PaTaRM. The upper part shows adaptive rubric generation for inference, while the lower part depicts the point-wise training procedure, where the dynamic rubric adaptation and Preference-Aware Reward (PAR) mechanism are incorporated into the reward modeling.
• Figure 3: Impact of different reward assignment functions $f(\cdot)$ under RL training on the RewardBench. $\Delta$ denotes the piecewise function, while $\alpha$ denotes the constant function.
• Figure 4: Performance of PaTaRM with voting@n on RewardBench.
• Figure 5: Primary rubric for the chat task.