V-Zero: Self-Improving Multimodal Reasoning with Zero Annotation

TL;DR

V-Zero targets the data-annotation bottleneck in multimodal reasoning by introducing a zero-annotation post-training framework that runs a co-evolutionary loop between a Questioner, $Q_ heta$, and a Solver, $S_ heta$, trained with Group Relative Policy Optimization ($GRPO$). The Questioner generates image-based MCQs and an intuitive answer, while the Solver answers using reasoning, with a dual-track reward encouraging deeper reflection and a formatting constraint to ensure clean data generation. Through difficulty-guided sampling and verifiable rewards, the Solver improves across general vision-centric and visual-mathematical benchmarks, often outperforming supervised baselines trained on annotated data. The results demonstrate the potential of internal feedback loops for advancing visual reasoning without human supervision, while noting compute and capacity limitations that guide future expansions to more architectures.

Abstract

Recent advances in multimodal learning have significantly enhanced the reasoning capabilities of vision-language models (VLMs). However, state-of-the-art approaches rely heavily on large-scale human-annotated datasets, which are costly and time-consuming to acquire. To overcome this limitation, we introduce V-Zero, a general post-training framework that facilitates self-improvement using exclusively unlabeled images. V-Zero establishes a co-evolutionary loop by instantiating two distinct roles: a Questioner and a Solver. The Questioner learns to synthesize high-quality, challenging questions by leveraging a dual-track reasoning reward that contrasts intuitive guesses with reasoned results. The Solver is optimized using pseudo-labels derived from majority voting over its own sampled responses. Both roles are trained iteratively via Group Relative Policy Optimization (GRPO), driving a cycle of mutual enhancement. Remarkably, without a single human annotation, V-Zero achieves consistent performance gains on Qwen2.5-VL-7B-Instruct, improving visual mathematical reasoning by +1.7 and general vision-centric by +2.6, demonstrating the potential of self-improvement in multimodal systems. Code is available at https://github.com/SatonoDia/V-Zero

Figures (3)

• Figure 1: Left: Comparison of training paradigms. Supervised GRPO (top) relies on static datasets where rewards are computed by comparing outputs against external ground-truth labels. In contrast, V-Zero (bottom) achieves self-improvement from unlabeled images via a dynamic co-evolutionary loop, where reward signals are internally generated through the interaction between a Questioner and a Solver. Right: Performance on Qwen2.5-VL-7B-Instruct. V-Zero demonstrates broad improvements across diverse benchmarks, notably outperforming both the Base Model and the strong Supervised GRPO baseline trained on human-labeled data.
• Figure 2: Overview of V-Zero. Our framework drives self-improvement through a co-evolutionary loop. Left (Questioner Training): The Questioner generates a question and an intuitive answer. A frozen Solver then samples reasoning responses to produce a pseudo-label via majority voting. The Questioner is updated via GRPO using a Dual-Track Reasoning Reward derived from the contrast between intuition and reasoning. Right (Solver Training): The optimized Questioner generates questions, which are paired with pseudo-labels from the Solver itself. This data undergoes difficulty-guided data sampling to filter for quality, and the Solver is then updated via RLVR.
• Figure 3: Improvement of generated questions. We visualize the questions generated by the Questioner for the same raw image (Left) across two iterations. In Iteration 1 (Middle), the model poses a straightforward area calculation problem. In Iteration 2 (Right), the question advances into a more complex ratio problem involving deeper geometric reasoning, demonstrating the progressive difficulty of the self-generated curriculum.