Self-Rewarding Vision-Language Model via Reasoning Decomposition

TL;DR

Vision-SR1 tackles visual hallucinations and language shortcuts in vision-language models by introducing a self-rewarding reinforcement learning framework that decomposes reasoning into visual perception and language reasoning. The method uses two-pass rollouts: an initial pass producing a self-contained visual perception and CoT, and a second pass that verifies the perception using only that perception to obtain a self-reward, which is combined with the final-answer reward. The approach avoids external supervision, improves grounding across diverse domains (general visual understanding, multimodal math, and hallucination diagnosis), and reduces reliance on language priors as evidenced by lower Language Shortcut Rates and stronger text-only reasoning in several benchmarks. A theoretical analysis argues that perception-level rewards increase the mutual information between visual input and final answers, guiding the model toward genuine visual grounding, while experiments on Vision-SR1-47K demonstrate consistent gains over strong baselines.

Abstract

Vision-Language Models (VLMs) often suffer from visual hallucinations, saying things that are not actually in the image, and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post-training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As a result, VLMs receive sparse visual signals and often learn to prioritize language-based reasoning over visual perception. To mitigate this, some existing methods add visual supervision using human annotations or distilled labels from external large models. However, human annotations are labor-intensive and costly, and because external signals cannot adapt to the evolving policy, they cause distributional shifts that can lead to reward hacking. In this paper, we introduce Vision-SR1, a self-rewarding method that improves visual reasoning without relying on external visual supervisions via reinforcement learning. Vision-SR1 decomposes VLM reasoning into two stages: visual perception and language reasoning. The model is first prompted to produce self-contained visual perceptions that are sufficient to answer the question without referring back the input image. To validate this self-containment, the same VLM model is then re-prompted to perform language reasoning using only the generated perception as input to compute reward. This self-reward is combined with supervision on final outputs, providing a balanced training signal that strengthens both visual perception and language reasoning. Our experiments demonstrate that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts across diverse vision-language tasks.

Figures (2)

• Figure 1: Overall framework of Vision-SR1. During RL training, the VLM performs two rollouts. In the first pass, the model takes an image–query pair and generates a structured output (visual perception, CoT reasoning, and answer), with answer reward computed against the ground truth. In the second pass, the model is re-prompted to answer using only query and its generated visual perception. If the correct answer is derived, a self-visual reward is assigned.
• Figure 2: We prompt Qwen-2.5-VL-7B to create the SFT cold-start dataset to learn the ideal format to get a quick start in the RL stage to reduce training steps and training time. Our cold start SFT dataset is the intersection of three source datasets, creating a final set of 9K examples. Our filtration process in Sec \ref{['sec:dataset']} guarantees zero false positives across text-only, caption-based, and direct CoT reasoning.