Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning

Zheng Zhang, Ao Lu, Yuanhao Zeng, Ziwei Shan, Jinjin Guo, Lufei Li, Yexin Li, Kan Ren

TL;DR

Grad2Reward addresses the challenge of sparse, sequence-level rewards in open-ended LLM reasoning by extracting dense token-level credit signals from the Judge's inference via a single backward pass. It employs gradient-based attribution to assign token-level contributions $r_t$ to the final verdict, enabling token-level GRPO without fine-tuning the Judge and incorporating a self-judging mechanism. The method demonstrates improved training efficiency and reasoning quality across multiple open-ended domains and model families, with robust performance across different test graders and strong cross-dataset generalization. The results also extend Grad2Reward to verifiable domains by integrating with process rewards, showing competitive performance against specialized reward models in mathematical reasoning tasks. Overall, Grad2Reward offers a scalable, general framework for dense supervision in open-ended and verifiable reasoning tasks, with implications for safer and more capable AI assistants.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant breakthroughs in complex LLM reasoning within verifiable domains, such as mathematics and programming. Recent efforts have sought to extend this paradigm to open-ended tasks by employing LLMs-as-a-Judge to provide sequence-level rewards for policy optimization. However, these rewards are inherently sparse, failing to provide the fine-grained supervision necessary for generating complex, long-form trajectories. Furthermore, current work treats the Judge as a black-box oracle, discarding the rich intermediate feedback signals encoded in it. To address these limitations, we introduce Grad2Reward, a novel framework that extracts dense process rewards directly from the Judge's model inference process via a single backward pass. By leveraging gradient-based attribution, Grad2Reward enables precise token-level credit assignment, substantially enhancing training efficiency and reasoning quality. Additionally, Grad2Reward introduces a self-judging mechanism, allowing the policy to improve through its own evaluative signals without training specialized reward models or reliance on superior external Judges. The experiments demonstrate that policies optimized with Grad2Reward achieve outstanding performance across diverse open-ended tasks, affirming its effectiveness and broad generalizability.

Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning

TL;DR

Grad2Reward addresses the challenge of sparse, sequence-level rewards in open-ended LLM reasoning by extracting dense token-level credit signals from the Judge's inference via a single backward pass. It employs gradient-based attribution to assign token-level contributions to the final verdict, enabling token-level GRPO without fine-tuning the Judge and incorporating a self-judging mechanism. The method demonstrates improved training efficiency and reasoning quality across multiple open-ended domains and model families, with robust performance across different test graders and strong cross-dataset generalization. The results also extend Grad2Reward to verifiable domains by integrating with process rewards, showing competitive performance against specialized reward models in mathematical reasoning tasks. Overall, Grad2Reward offers a scalable, general framework for dense supervision in open-ended and verifiable reasoning tasks, with implications for safer and more capable AI assistants.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant breakthroughs in complex LLM reasoning within verifiable domains, such as mathematics and programming. Recent efforts have sought to extend this paradigm to open-ended tasks by employing LLMs-as-a-Judge to provide sequence-level rewards for policy optimization. However, these rewards are inherently sparse, failing to provide the fine-grained supervision necessary for generating complex, long-form trajectories. Furthermore, current work treats the Judge as a black-box oracle, discarding the rich intermediate feedback signals encoded in it. To address these limitations, we introduce Grad2Reward, a novel framework that extracts dense process rewards directly from the Judge's model inference process via a single backward pass. By leveraging gradient-based attribution, Grad2Reward enables precise token-level credit assignment, substantially enhancing training efficiency and reasoning quality. Additionally, Grad2Reward introduces a self-judging mechanism, allowing the policy to improve through its own evaluative signals without training specialized reward models or reliance on superior external Judges. The experiments demonstrate that policies optimized with Grad2Reward achieve outstanding performance across diverse open-ended tasks, affirming its effectiveness and broad generalizability.
Paper Structure (48 sections, 15 equations, 2 figures, 6 tables, 2 algorithms)

This paper contains 48 sections, 15 equations, 2 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM for Open-ended Tasks.
  4. Dense Rewards Modeling.
  5. Preliminary
  6. Open-ended LLM Reasoning
  7. LLM-as-a-Judge
  8. Methodology
  9. Judge Implicitly Contains Process Feedback
  10. Self-Judging Mechanism.
  11. Fine-Grained Reward Design
  12. Theoretical Analysis of the Reward Design
  13. Policy Optimization via Token-level GRPO
  14. Experiments
  15. Experimental Settings
  16. ...and 33 more sections

Figures (2)

  • Figure 1: Overview of Grad2Reward (better viewed in color): The Policy samples an output $o$, which is evaluated by a Judge to derive a verdict $z$. By computing the inner product between the output embeddings $e_t$ and their gradients $g_t$ (derived from $z$), we obtain token-wise attribution scores $b_t$. These are converted into token-level rewards $r_t$ to guide the policy optimization.
  • Figure 2: Training dynamics on (a) HealthBench, (b) RaR-Medicine with Qwen3-30B-A3B-Instrcut as test grader, and (c) HealthBench, (d) RaR-Medicine with Mistral-Small-3.2-24B-Instruct as test grader.