Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards

TL;DR

<3-5 sentence high-level summary> This survey addresses the problem of aligning large language models with human values and task objectives through learning from rewards. It presents a unified framework linking language models, reward models, and learning strategies across training, inference, and post-inference, and surveys scalar, critique, implicit, rule-based, and process rewards, including both human and automated feedback. Key contributions include a detailed taxonomy, analysis of benchmarking efforts, and discussion of applications such as preference alignment and mathematical reasoning, along with challenges like interpretability, reward hacking, and continual learning. The work highlights the potential of reward-driven approaches to enable robust, scalable, and agentic AI capable of operating in dynamic real-world settings.

Abstract

Recent developments in Large Language Models (LLMs) have shifted from pre-training scaling to post-training and test-time scaling. Across these developments, a key unified paradigm has arisen: Learning from Rewards, where reward signals act as the guiding stars to steer LLM behavior. It has underpinned a wide range of prevalent techniques, such as reinforcement learning (RLHF, RLAIF, DPO, and GRPO), reward-guided decoding, and post-hoc correction. Crucially, this paradigm enables the transition from passive learning from static data to active learning from dynamic feedback. This endows LLMs with aligned preferences and deep reasoning capabilities for diverse tasks. In this survey, we present a comprehensive overview of learning from rewards, from the perspective of reward models and learning strategies across training, inference, and post-inference stages. We further discuss the benchmarks for reward models and the primary applications. Finally we highlight the challenges and future directions. We maintain a paper collection at https://github.com/bobxwu/learning-from-rewards-llm-papers.

Figures (12)

• Figure 1: Illustration of the scaling phases of LLMs. The learning-from-rewards paradigm plays a pivotal role in the post-training and test-time scaling.
• Figure 2: A unified framework of learning from rewards. The language model generates outputs conditioned on the inputs; the reward model evaluates the outputs and provides reward signals based on diverse feedback sources and design choices; the learning strategy leverages the rewards to either fine-tune the language model or refine the outputs. This learning-from-rewards paradigm aims to fulfill preference alignment and task-specific goals. The learning strategy can occur at the training, inference, or post-inference stages.
• Figure 3: Reward Model (RM) design dimensions: (a) Base Architecture (Model-based and Model-free); (b) Reward Format (Scalar, Critique, and Implicit); (c) Scoring Pattern (Pointwise and Pairwise); (d) Reward Granularity (Outcome and Process).
• Figure 4: Illustration of Training with Rewards. Based on the reward model design, we mainly focus on scalar rewards, critique rewards, implicit rewards, rule-based rewards, and process rewards. These reward signals are used to fine-tune the language model through reinforcement learning algorithms or supervised fine-tuning.
• Figure 5: Overview of Training with Rewards.