Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

TL;DR

<3-5 sentence high-level summary> RR VF proposes a self-supervised reinforcement learning framework that enables multimodal language models to learn complex visual reasoning purely from raw images by iterating reasoning, rendering, and visual feedback, guided by the Asymmetry of Verification. It employs GRPO for stable optimization and a hybrid reward design that blends visual similarity, format correctness, and tool-use incentives. Across chart-to-code and web-to-code tasks, RR VF achieves state-of-the-art open-source performance for models of similar size and demonstrates strong generalization to unseen datasets without image-text supervision. This work suggests a scalable path for advancing visual reasoning in MLLMs by exploiting pixel-level feedback rather than textual supervision.

Abstract

Multimodal Large Language Models (MLLMs) exhibit impressive performance across various visual tasks. Subsequent investigations into enhancing their visual reasoning abilities have significantly expanded their performance envelope. However, a critical bottleneck in the advancement of MLLMs toward deep visual reasoning is their heavy reliance on curated image-text supervision. To solve this problem, we introduce a novel framework, ``Reasoning-Rendering-Visual-Feedback'' (RRVF), that enables MLLMs to learn complex visual reasoning from only raw images. This framework builds on the ``Asymmetry of Verification'' principle, i.e., verifying the rendered output against the source image is substantially easier than performing deep visual reasoning to generate a faithful, structured representation such as code. We demonstrate that this relative ease provides an ideal reward signal for optimization via Reinforcement Learning (RL), thereby reducing reliance on image-text supervision. RRVF implements a closed-loop iterative process encompassing reasoning, rendering, and visual feedback components, enabling the model to perform complex reasoning, including self-correction through multi-turn interactions. This process is optimized end-to-end using the GRPO algorithm. Extensive evaluations are conducted on image-to-code generation across two diverse domains: data charts and web interfaces. The RRVF-trained model not only outperforms existing similarly sized open-source MLLMs and supervised fine-tuning baselines but also exhibits superior generalization. Notably, the model outperforms the more advanced MLLM used to generate visual feedback during training. Code is available at https://github.com/L-O-I/RRVF.

Figures (10)

• Figure 1: Comparison of training paradigms. Conventional MLLMs (top) rely on paired image-text data. Our RRVF (bottom) trains solely on raw images via a closed-loop reasoning-rendering-visual-feedback process under GRPO.
• Figure 2: RRVF is a training framework that improves visual reasoning ability using only image inputs. Given an input image (a), the multimodal language model performs iterative reasoning (b) by generating rendering code. The code is executed by external tools, and the output is compared with the original image by a visual judge (d). The comparison results are converted into structured feedback (c), which guides the next round of code generation. The reward manager (e) combines visual similarity, format validity, and tool usage efficiency. These signals are used to optimize the model through reinforcement learning, implemented with the GRPO algorithm.
• Figure 3: Training curves for reward score and tool usage.
• Figure 4: An illustration of the RRVF training process. The code for a pie chart is iteratively refined based on visual feedback. (Turn 1) A basic pie chart is generated but lacks the required exploded slice effect. (Turn 2) Following feedback, the code is updated to incorporate the effect and a legend, though color discrepancies persist. (Turn 3) The color palette and legend are adjusted to align with the reference image. The progressive refinement is reflected in the increasing score (75 $\to$ 83 $\to$ 85), demonstrating the efficacy of the closed-loop learning process.
• Figure 5: Training examples illustrating the generalizability of the RRVF framework. The framework demonstrates its capability to iteratively refine diverse chart types, including bar, line, and scatter plots. This is achieved by progressively correcting errors in structure, data, and style based on multi-turn visual feedback.