VisPlay: Self-Evolving Vision-Language Models from Images

TL;DR

<pVisPlay introduces a self-evolving reinforcement learning framework for Vision-Language Models by decomposing a base model into an Image-Conditioned Questioner and a Multimodal Reasoner, trained with Group Relative Policy Optimization to balance question difficulty and answer quality without human supervision. The two roles co-evolve through a closed-loop pipeline, using pseudo-labels and uncertainty/diversity rewards to progressively enhance visual reasoning and reduce hallucinations across multiple backbones and eight benchmarks. Empirical results show strong, consistent gains across general understanding, visual mathematics, and hallucination detection, with competitive performance relative to models trained on human-annotated data, and clear evidence of co-evolution dynamics and difficulty progression. VisPlay demonstrates a scalable path toward autonomous multimodal intelligence that continually improves from unlabeled image data, reducing reliance on manual annotation while delivering robust reasoning capabilities.>

Abstract

Reinforcement learning (RL) provides a principled framework for improving Vision-Language Models (VLMs) on complex reasoning tasks. However, existing RL approaches often rely on human-annotated labels or task-specific heuristics to define verifiable rewards, both of which are costly and difficult to scale. We introduce VisPlay, a self-evolving RL framework that enables VLMs to autonomously improve their reasoning abilities using large amounts of unlabeled image data. Starting from a single base VLM, VisPlay assigns the model into two interacting roles: an Image-Conditioned Questioner that formulates challenging yet answerable visual questions, and a Multimodal Reasoner that generates silver responses. These roles are jointly trained with Group Relative Policy Optimization (GRPO), which incorporates diversity and difficulty rewards to balance the complexity of generated questions with the quality of the silver answers. VisPlay scales efficiently across two model families. When trained on Qwen2.5-VL and MiMo-VL, VisPlay achieves consistent improvements in visual reasoning, compositional generalization, and hallucination reduction across eight benchmarks, including MM-Vet and MMMU, demonstrating a scalable path toward self-evolving multimodal intelligence. The project page is available at https://bruno686.github.io/VisPlay/

Figures (3)

• Figure 1: Illustration of the average accuracy improvement (averaged over seven datasets) through successive evolutions (Evo 1 to Evo 5) on Qwen2.5-VL-3B-Instruct, compared to a baseline trained on Vision-47K with GRPO, demonstrating the effectiveness of our VisPlay .
• Figure 2: An illustration of our VisPlay framework, depicting the co-evolution of the Image-Conditioned Questioner and Multimodal Reasoner. Top: During the Questioner training stage, the Image-Conditioned Questioner is optimized via GRPO to produce challenge questions. The reward stems from the uncertainty of the frozen Multimodal Reasoner, computed by the consistency of its multiple generated answers. Bottom: In the Reasoner training stage, the Multimodal Reasoner is trained via GRPO on a curated set of challenging questions from the now-frozen Image-Conditioned Questioner, leveraging pseudo-labels from its own majority voting.
• Figure 3: Changes in question difficulty (orange, left axis) and problem-solving accuracy (blue, right axis) during Image-Conditioned Questioner and Multimodal Reasoner training across three VLMs.