Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains

TL;DR

SR-MCR addresses the core problem of unreliable intermediate reasoning in multimodal LLMs by introducing a label-free, process-aware alignment framework. It fuses five intrinsic self-signals—semantic alignment, lexical fidelity, non-redundancy, visual grounding, and step-wise coherence—into a normalized self-reward $R(I,x,\hat{y}_a,\hat{y}_t)$ with adaptive reliability weights, and optimizes via a critic-free GRPO with a cooling mechanism to suppress noisy or overconfident updates. The approach, built on Qwen2.5-VL and implemented with LoRA adapters, yields consistent improvements across general multimodal benchmarks and state-of-the-art performance among open-source 7B models (average 81.4%); it also achieves strong V*Bench results, demonstrating robust grounding and spatial reasoning. Ablation and cross-architecture studies show the five rewards are complementary, the cooling module stabilizes training, and external reward signals provide reliable grounding better than LLM-based judges. Overall, SR-MCR offers a scalable, noise-robust, and label-free pathway to enhance accuracy, grounding, and coherence in diverse visual domains, with broad applicability to other multimodal architectures.

Abstract

Multimodal LLMs often produce fluent yet unreliable reasoning, exhibiting weak step-to-step coherence and insufficient visual grounding, largely because existing alignment approaches supervise only the final answer while ignoring the reliability of the intermediate reasoning process. We introduce SR-MCR, a lightweight and label-free framework that aligns reasoning by exploiting intrinsic process signals derived directly from model outputs. Five self-referential cues -- semantic alignment, lexical fidelity, non-redundancy, visual grounding, and step consistency -- are integrated into a normalized, reliability-weighted reward that provides fine-grained process-level guidance. A critic-free GRPO objective, enhanced with a confidence-aware cooling mechanism, further stabilizes training and suppresses trivial or overly confident generations. Built on Qwen2.5-VL, SR-MCR improves both answer accuracy and reasoning coherence across a broad set of visual benchmarks; among open-source models of comparable size, SR-MCR-7B achieves state-of-the-art performance with an average accuracy of 81.4%. Ablation studies confirm the independent contributions of each reward term and the cooling module.

Figures (7)

• Figure 1: Overview of SR-MCR. Given an image--text input $(I,x)$, the policy $\pi_\theta$ generates multiple responses, each scored by five process-level self-reward terms and an adaptive reliability estimator to form a mixed reward. SR-GRPO then favors high-reward, reliable outputs while suppressing trivial ones, updating $\pi_\theta$ on top of the frozen base policy $\pi_0$.
• Figure 2: Training pipeline of SR-MCR. We load data and a pretrained VLM, sample $K$ responses via vLLM kwon2023efficient, compute component-wise self-rewards, and train a LoRA adapter with the SR-GRPO loss. Rewards and hyperparameters are specified in a single YAML configuration.
• Figure 3: MME ACC+ performance at different training steps across four task types. ACC+ is counted only when both QA pairs for an image are correctly answered.
• Figure 4: Ablation on Optimization Strategy. SR-GRPO (cooling) vs. PPO and no-cooling. Avg. 3B/7B scores show both GRPO and cooling are needed.
• Figure 5: Reasoning quality evaluation. On 100 A/B samples, SR-MCR-7B is preferred (90.82%) over the Qwen2.5-VL-7B baseline (9.18%).