Reinforcement Learning for Large Model: A Survey

TL;DR

The paper surveys the rapid expansion of visual reinforcement learning for multimodal large models, organizing over 200 works into four pillars: multimodal LLMs, visual generation, unified frameworks, and vision-language-action agents. It synthesizes three core alignment paradigms (RLHF, DPO, RLVR) and two key policy-optimization algorithms (PPO and GRPO), highlighting how group-relative signals and verifiable rewards improve stability and scalability. It catalogs metrics and benchmarks across task-level, reward-level, and training-state evaluations, and identifies persistent challenges in sample efficiency, generalization, and safe deployment. By providing a principled taxonomy and actionable insights on reward design, curriculum strategies, and evaluation standards, the survey aims to guide researchers toward more efficient, reliable, and societally aligned visual RL systems.

Abstract

Recent advances at the intersection of reinforcement learning (RL) and visual intelligence have enabled agents that not only perceive complex visual scenes but also reason, generate, and act within them. This survey offers a critical and up-to-date synthesis of the field. We first formalize visual RL problems and trace the evolution of policy-optimization strategies from RLHF to verifiable reward paradigms, and from Proximal Policy Optimization to Group Relative Policy Optimization. We then organize more than 200 representative works into four thematic pillars: multi-modal large language models, visual generation, unified model frameworks, and vision-language-action models. For each pillar we examine algorithmic design, reward engineering, benchmark progress, and we distill trends such as curriculum-driven training, preference-aligned diffusion, and unified reward modeling. Finally, we review evaluation protocols spanning set-level fidelity, sample-level preference, and state-level stability, and we identify open challenges that include sample efficiency, generalization, and safe deployment. Our goal is to provide researchers and practitioners with a coherent map of the rapidly expanding landscape of visual RL and to highlight promising directions for future inquiry. Resources are available at: https://github.com/weijiawu/Awesome-Visual-Reinforcement-Learning.

Figures (6)

• Figure 1: Timeline of Representative Visual Reinforcement Learning Models. The figure presents a chronological overview of key Visual RL models from 2023 to 2025, organized into four tracks: Multimodal LLM, Visual Generation, Unified Models, and VLA Models.
• Figure 2: Three Alignment Paradigms for Reinforcement Learning. (a) RLHF learns a reward model from human preference data and optimizes the policy via PPO. (b) DPO removes the reward model and directly optimizes a contrastive objective against a frozen reference. (c) RLVR replaces subjective preferences with deterministic verifiable signals and trains the policy using GRPO.
• Figure 3: Two Representative Policy Optimization Algorithms for LLM. PPO (a) uses a learned value model $V_\psi$ for advantage estimation and injects the KL penalty at each token. GRPO (b) removes the value model, computes group-normalized advantages $\hat{A}_{i,t}$ across $G$ continuations, and applies an explicit prompt-level KL penalty.
• Figure 4: Overall taxonomy of reinforcement-learning research in vision. The chart groups existing work by high-level domain (MLLMs, visual generation, unified models, and vision-language action agents) and then by finer-grained tasks, illustrating representative papers for each branch.
• Figure 5: Three reward paradigms for RL-based image generation. (a) Human-Centric Preference Optimization: aligns outputs with human aesthetic scores (HPS wu2023human, ImageReward xu2023imagereward); (b) Multimodal Reasoning-Based Evaluation: scores images via multimodal reasoning consistency (UnifiedReward wang2025unified, PARM guo2025can); (c) Metric-Driven Objective Optimization: minimizes task-specific quantitative metrics such as FID and IoU.