AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling

TL;DR

AdaJudge tackles the rigidity of static pooling in reward modeling by introducing a two-stage approach that (i) refines backbone representations through depth-gated blocks and (ii) employs a domain-aware, multi-view pooling head whose routing adapts to the prompt. This adaptive, gated mixture of views enables the model to capture both localized and global evidence appropriate to each task, improving discrimination across heterogeneous domains. Empirical results on RM-Bench and JudgeBench show AdaJudge attaining state-of-the-art performance, with notable gains on hard cases and compact backbones, and ablations confirm the value of iterative refinement and adaptive aggregation. The work advances reward modeling by aligning representation and aggregation with task-specific evaluation signals, offering practical improvements for aligning LLMs to human preferences with greater reliability and efficiency.

Abstract

Reward modeling is essential for aligning large language models with human preferences, yet predominant architectures rely on a static pooling strategy to condense sequences into scalar scores. This paradigm, however, suffers from two key limitations: a static inductive bias that misaligns with task-dependent preference signals, and a representational mismatch, as the backbone is optimized for generation rather than fine-grained discrimination. To address this, we propose AdaJudge, a unified framework that jointly adapts representation and aggregation. AdaJudge first refines backbone representations into a discrimination-oriented space via gated refinement blocks. It then replaces the static readout with an adaptive multi-view pooling module that dynamically routes and combines evidence. Extensive experiments on RM-Bench and JudgeBench show that AdaJudge outperforms strong off-the-shelf reward models and traditional pooling baselines.

Figures (4)

• Figure 1: Task-dependent biases of static pooling strategies: distinct pooling mechanisms show conflicting biases: Last-token pooling excels at terminal Math errors ("62") but misses Safety violations masked by safe endings ("don't recommend"). Conversely, Mean pooling captures holistic harm but is diluted by correct reasoning steps in Math. This inductive bias mismatch limits static strategies across heterogeneous tasks.
• Figure 2: The illustration of AdaJudge, a two-stage reward modeling framework. Given a prompt--response pair, an LLM backbone produces token-level representations $\mathrm{{H}^{(0)}}$. Stage I Adaptive Representation Refinement applies $\mathrm{K}$ lightweight refinement blocks and a depth router to adaptively combine intermediate states into a refined representation $\mathrm{H}^{(0)}$. Stage II Multi-Perspective Aggregation extracts three complementary response features from $\mathrm{\tilde{H}}$ via last-token, mean, and attention pooling, producing $z_L$, $z_M$, and $z_A$, while mean pooling over prompt tokens yields a prompt context $z_P$. Each feature is mapped to a scalar score by an MLP head, and a pooling router conditioned on $[z_L; z_M; z_A; z_P]$ predicts routing weights $\pi$ to form the final reward $r$ as a gated mixture of perspective scores (shown by $\oplus$).
• Figure 3: Gate-weighted alignment before and after Stage-I refinement on RM-Bench (left) and JudgeBench (right). Alignment is measured as the cosine similarity between the scoring direction $\nabla_{\mathbf{z}} r(\mathbf{x},\mathbf{y})$ and the preference difference $\mathbf{z}(x^+)-\mathbf{z}(x^-)$, aggregated using AdaJudge’s routing weights.
• Figure 4: Average routing weights of AdaJudge’s pooling experts across domains on RM-Bench (left) and JudgeBench (right). For each benchmark, we group pairwise preference samples by domain and compute the mean gating weights assigned to the Last, Attention, and Mean pooling experts over the responses, using a Qwen3-4B–based AdaJudge reward model.