Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

TL;DR

This work reframes RL-based fine-tuning of LLMs as Reward Modeling for Reasoning, arguing that reward design is a central driver of reasoning quality, generalization, and trust. It introduces Reasoning-Aligned Reinforcement Learning (RARL) to unify model-based, rule-based, and self-reward paradigms, and provides a structured taxonomy of reward architectures, granularities, and semantics. The paper analyzes reward hacking and system-level biases (credit assignment, distribution shift, length, position, and faithfulness) and discusses strategies to mitigate them, including online learning, ensembles, and retrieval-grounded approaches. It extends reward design to inference-time planning, bias mitigation, and augmented reasoning (RAG, tools, and tables), and reviews evaluation benchmarks, containment challenges, and domain-specific applications in medicine, finance, and science, offering future directions toward robust, verifiable, and domain-adaptive RL-based reasoning models.

Abstract

Large Language Models (LLMs) demonstrate transformative potential, yet their reasoning remains inconsistent and unreliable. Reinforcement learning (RL)-based fine-tuning is a key mechanism for improvement, but its effectiveness is fundamentally governed by reward design. Despite its importance, the relationship between reward modeling and core LLM challenges--such as evaluation bias, hallucination, distribution shift, and efficient learning--remains poorly understood. This work argues that reward modeling is not merely an implementation detail but a central architect of reasoning alignment, shaping what models learn, how they generalize, and whether their outputs can be trusted. We introduce Reasoning-Aligned Reinforcement Learning (RARL), a unifying framework that systematizes diverse reward paradigms for multi-step reasoning. Within this framework, we present a taxonomy of reward mechanisms, analyze reward hacking as a pervasive failure mode, and examine how reward signals unify challenges ranging from inference-time scaling to hallucination mitigation. We further critically evaluate existing benchmarks, highlighting vulnerabilities such as data contamination and reward misalignment, and outline directions for more robust evaluation. By integrating fragmented research threads and clarifying the interplay between reward design and fundamental reasoning capabilities, this work provides a foundational roadmap for building reasoning models that are robust, verifiable, and trustworthy.

Figures (3)

• Figure 1: Overview of the work. We first introduce reward design in RL (Section \ref{['sec:design']}) and identify key challenges associated with reward hacking (Section \ref{['sec:hack']}). We then show how reward signals can serve as a unified mechanism for improving LLM inference-time reasoning and efficiency (Section \ref{['sec:usecase']}), mitigating LLM bias (Section \ref{['sec:LLM-bias']}), enabling robust augmented reasoning (Section \ref{['sec:augmented']}), and broader reinforcement learning challenges (Section \ref{['sec:RL-issue']}). We also discuss reward metrics and benchmarks for text-only and multimodal reward models, and key traits to build a successful benchmark (Section \ref{['sec:evaluation']}), followed by the investigation of the reward design in real-world applications (Section \ref{['sec:app']}).
• Figure 2: The unification of existing popular frameworks for RL fine-tuning.
• Figure 3: A taxonomy of reward designs for large reasoning language models.