Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models

Haoyuan Sun, Jiaqi Wu, Bo Xia, Yifu Luo, Yifei Zhao, Kai Qin, Xufei Lv, Tiantian Zhang, Yongzhe Chang, Xueqian Wang

TL;DR

This work addresses the challenge of enabling robust reasoning in multimodal large language models by positioning reinforcement fine-tuning (RFT) as a unifying training paradigm. It develops a taxonomy that distinguishes Critic-Model-Driven from Critic-Model-Free RFT and surveys how PPO-style policy optimization is adapted for multimodal settings, supported by $V_\pi(s)$ and $Q_\pi(s,a)$ foundations. The paper identifies five broad success areas—modality diversity, task/domain breadth, improved training algorithms, abundant benchmarks, and practical engineering frameworks—and provides five concrete future directions, including generalization, hybrid reward paradigms, safety, data augmentation, and cross-domain applications. Taken together, the insights map a clear path toward more capable, generalizable, and safe RFT-enabled reasoning systems for MLLMs, with implications for the broader push toward AGI-era multimodal reasoning. The emphasis on practical frameworks and benchmarks also offers actionable guidance for researchers and practitioners deploying RFT in real-world multimodal contexts.

Abstract

Standing in 2025, at a critical juncture in the pursuit of Artificial General Intelligence (AGI), reinforcement fine-tuning (RFT) has demonstrated significant potential in enhancing the reasoning capability of large language models (LLMs) and has led to the development of cutting-edge AI models such as OpenAI-o1 and DeepSeek-R1. Moreover, the efficient application of RFT to enhance the reasoning capability of multimodal large language models (MLLMs) has attracted widespread attention from the community. In this position paper, we argue that reinforcement fine-tuning powers the reasoning capability of multimodal large language models. To begin with, we provide a detailed introduction to the fundamental background knowledge that researchers interested in this field should be familiar with. Furthermore, we meticulously summarize the improvements of RFT in powering reasoning capability of MLLMs into five key points: diverse modalities, diverse tasks and domains, better training algorithms, abundant benchmarks and thriving engineering frameworks. Finally, we propose five promising directions for future research that the community might consider. We hope that this position paper will provide valuable insights to the community at this pivotal stage in the advancement toward AGI. Summary of works done on RFT for MLLMs is available at https://github.com/Sun-Haoyuan23/Awesome-RL-based-Reasoning-MLLMs.

Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models

TL;DR

This work addresses the challenge of enabling robust reasoning in multimodal large language models by positioning reinforcement fine-tuning (RFT) as a unifying training paradigm. It develops a taxonomy that distinguishes Critic-Model-Driven from Critic-Model-Free RFT and surveys how PPO-style policy optimization is adapted for multimodal settings, supported by and foundations. The paper identifies five broad success areas—modality diversity, task/domain breadth, improved training algorithms, abundant benchmarks, and practical engineering frameworks—and provides five concrete future directions, including generalization, hybrid reward paradigms, safety, data augmentation, and cross-domain applications. Taken together, the insights map a clear path toward more capable, generalizable, and safe RFT-enabled reasoning systems for MLLMs, with implications for the broader push toward AGI-era multimodal reasoning. The emphasis on practical frameworks and benchmarks also offers actionable guidance for researchers and practitioners deploying RFT in real-world multimodal contexts.

Abstract

Standing in 2025, at a critical juncture in the pursuit of Artificial General Intelligence (AGI), reinforcement fine-tuning (RFT) has demonstrated significant potential in enhancing the reasoning capability of large language models (LLMs) and has led to the development of cutting-edge AI models such as OpenAI-o1 and DeepSeek-R1. Moreover, the efficient application of RFT to enhance the reasoning capability of multimodal large language models (MLLMs) has attracted widespread attention from the community. In this position paper, we argue that reinforcement fine-tuning powers the reasoning capability of multimodal large language models. To begin with, we provide a detailed introduction to the fundamental background knowledge that researchers interested in this field should be familiar with. Furthermore, we meticulously summarize the improvements of RFT in powering reasoning capability of MLLMs into five key points: diverse modalities, diverse tasks and domains, better training algorithms, abundant benchmarks and thriving engineering frameworks. Finally, we propose five promising directions for future research that the community might consider. We hope that this position paper will provide valuable insights to the community at this pivotal stage in the advancement toward AGI. Summary of works done on RFT for MLLMs is available at https://github.com/Sun-Haoyuan23/Awesome-RL-based-Reasoning-MLLMs.

Paper Structure

This paper contains 19 sections, 3 theorems, 7 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning: Value-based and Policy-based Methods
  4. Multimodal Reasoning: Language-Centric and Collaborative Paradigms
  5. Reinforcement Fine-Tuning: Critic-Model-Driven and Critic-Model-Free Algorithms
  6. RFT for MLLMs: What has the community done?
  7. Future Work: What could the community do next?
  8. Summary of works done on RFT for MLLMs
  9. Vision (Image)
  10. Vision (Video)
  11. Medical Vision
  12. Embodied Vision
  13. Multimodal Reward Model
  14. Audio
  15. Omni
  16. ...and 4 more sections

Key Result

Proposition 2.1

The current policy, parameterized by $\theta_{\text{old}}$, is denoted as $\pi_{\theta_{\text{old}}}$. Primary goal is to find a better policy $\pi_{\theta}$ utilizing current policy $\pi_{\theta_{\text{old}}}$. The objective is: where $\upsilon^{\pi}$ is the state visitation distribution under policy $\pi$; $A_\pi(s,a)$ is the advantage function with definition of $A_\pi(s,a)=Q_{\pi}(s,a)-V_{\pi

Figures (1)

  • Figure 1: An overview of works done on reinforcement fine-tuning (RFT) for multimodal large language models (MLLMs). Works are sorted by release time and are collected up to May 15, 2025. Further detailed summary is provided in \ref{['Summary of works done on RFT for MLLMs']}.

Theorems & Definitions (5)

  • Definition 2.1: State Value Function sutton1998reinforcement
  • Definition 2.2: Action Value Function sutton1998reinforcement
  • Proposition 2.1: TRPO Objective schulman2015trust
  • Proposition 2.2: PPO Objectives schulman2017proximal
  • Proposition 2.3: GRPO Objective guo2025deepseekshao2024deepseekmath