NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation

TL;DR

NoisyGRPO addresses generalization gaps in multimodal CoT reasoning by coupling noise-injected visual exploration with a Bayesian advantage Estimation that fuses a noise prior and trajectory rewards. The method introduces a diffusion-based noise strategy during rollout collection and derives a posterior trajectory quality to guide policy updates via a PPO-style surrogate objective with clipping and KL regularization. Empirical results on CoT quality, general capability, and hallucination benchmarks show substantial gains over GRPO, especially for small-scale MLLMs such as Qwen2.5-VL 3B, while maintaining training efficiency. The framework demonstrates robust performance across diverse tasks and datasets, highlighting improved visual-grounded reasoning and better out-of-distribution generalization in multimodal settings.

Abstract

Reinforcement learning (RL) has shown promise in enhancing the general Chain-of-Thought (CoT) reasoning capabilities of multimodal large language models (MLLMs). However, when applied to improve general CoT reasoning, existing RL frameworks often struggle to generalize beyond the training distribution. To address this, we propose NoisyGRPO, a systematic multimodal RL framework that introduces controllable noise into visual inputs for enhanced exploration and explicitly models the advantage estimation process via a Bayesian framework. Specifically, NoisyGRPO improves RL training by: (1) Noise-Injected Exploration Policy: Perturbing visual inputs with Gaussian noise to encourage exploration across a wider range of visual scenarios; and (2) Bayesian Advantage Estimation: Formulating advantage estimation as a principled Bayesian inference problem, where the injected noise level serves as a prior and the observed trajectory reward as the likelihood. This Bayesian modeling fuses both sources of information to compute a robust posterior estimate of trajectory advantage, effectively guiding MLLMs to prefer visually grounded trajectories over noisy ones. Experiments on standard CoT quality, general capability, and hallucination benchmarks demonstrate that NoisyGRPO substantially improves generalization and robustness, especially in RL settings with small-scale MLLMs such as Qwen2.5-VL 3B. The project page is available at https://artanic30.github.io/project_pages/NoisyGRPO/.

Figures (13)

• Figure 1: Performance and training statistics of three RL methods.GRPO with noise injection refers to GRPO trained with noise-perturbed rollouts. The left plot shows evaluation performance on the MMStar benchmark over training iterations. The middle plot presents the standard deviation of rewards, reflecting the exploration degree of the policy. The right plot shows the accuracy reward, indicating how well the model fits the training data. Shaded areas represent the variance of the corresponding metrics.
• Figure 2: Overall pipeline of NoisyGRPO. For each image-question pair, we sample noise and inject it into the image. The policy model generates rollouts based on the perturbed inputs, and the reward function evaluates them. We then compute the posterior advantage by combining the noise-based prior with the reward-based observation.
• Figure 3: Performance over iteration and training statistics. We report the evaluation results of NoisyGRPO-3B on the MMStar benchmark to demonstrate the comprehensive capabilities of the MLLM. For both Importance Weight and Completion Length, the shaded regions represent the variance across samples.
• Figure 4: Performance of Ablation Study. We report the average performance on the MMStar benchmark across the following ablation settings: (a) Ablation of core design choices in NoisyGRPO; (b) Sensitivity to the hyperparameter $\alpha$ controlling observation confidence; (c) Sensitivity to the hyperparameter $\gamma$ modulating prior variance adaptation; (d) Comparison of different noise distributions used in the noise-injected exploration policy.
• Figure 5: Correlation of correctness and noise level.