Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling

TL;DR

The paper addresses limitations of scalar reward models and black-box generative rewards in industrial settings by proposing the Structural Reward Model (SRM), which uses Side-Branch Models (SBMs) to generate interpretable auxiliary features that enrich reward evaluation. SBMs (Semantic, Entity, Fact-Check, Style, Quality) are trained via Best-of-N sampling and LLM-as-judge to produce high-quality auxiliary texts, which are concatenated with the main prompt-response pairs and scored by a lightweight RM under a Bradley-Terry loss; key equations include $L_{SB_i}(\phi_i)= -\frac{1}{|\mathcal{D}_{auxiliary}|} \sum_{(p,r,a)\in \mathcal{D}_{auxiliary}} \log P_{\phi_i}(a\mid p,r)$, $P(r_c \succ r_j | p)=\frac{e^{s_c}}{e^{s_c}+e^{s_j}}$, with $s_c=RM(x_{chosen};\theta)$ and $s_j=RM(x_{reject};\theta)$, and $\mathcal{L}_{BT-RM}(\theta)=-\frac{1}{|\mathcal{D}_{t}|}\sum_{(p,r_c,r_j)\in\mathcal{D}_{t}}\log P(r_c \succ r_j | p)$. The approach demonstrates robust, consistent improvements across RM-Bench, JudgeBench, and IFBench, with ablations highlighting the crucial roles of Fact-Checking and Semantic Understanding. This modular, parallelizable design offers dimension-aware diagnostics and scalable deployment for industrial LLM systems, though it relies on domain-specific SBMs, high-quality data, and careful fusion of auxiliary features.

Abstract

Reward Models (RMs) are key components for evaluating and guiding language model outputs. However, traditional scalar RMs often struggle with incorporating contextual and background information during inference, leading to incomplete evaluations. Generative RMs (GRMs) attempt to address these limitations by generating intermediate reasoning steps. Yet, their uncontrolled black-box nature and inefficiency due to sequential decoding hinder their industrial deployment. Industrial scenarios, such as search and recommendation systems, often involve single-domain tasks requiring evaluation along specific dimensions. In such contexts, diagnosing "bad cases" necessitates structured feedback to identify and optimize dimension-specific issues. In this paper, we propose the Structural Reward Model (SRM), a modular and interpretable framework integrating side-branch models as auxiliary feature generators. By introducing fine-grained dimensions, SRMs enable interpretable and efficient evaluation, facilitating targeted diagnostics and optimization. This structured approach ensures adaptability and scalability for industrial applications. Through comprehensive experiments, we demonstrate that SRMs outperform scalar RMs and GRMs in robustness and alignment with human preferences. The modular design further supports efficient optimization for practical scenarios, allowing SRM to provide a practical reward modeling solution for industry.

Figures (3)

• Figure 1: Overview of the Side Branch Models Enhanced Structural Reward Model architecture.
• Figure 2: Overview of the proposed Enhanced Structural Reward Model(SRM) framework integrated with Side Branch Models (SBMs). Given a prompt and candidate responses, the SBMs (Semantic Understanding, Entity Expansion, Fact-Checking, Style Matching, Quality Assessment) generate auxiliary textual contexts. These contexts augment the original input pairs, enabling the Reward Model to deliver evaluations with enhanced accuracy, robustness, and alignment with human preferences.
• Figure 3: Ablation study of side branch models (%)