Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme

TL;DR

This work tackles reproducibility and evaluation gaps in reinforcement learning for vision-language models by introducing Maye, a transparent from-scratch four-step RL framework, and a standardized evaluation scheme to track training dynamics and reflective behavior. The approach emphasizes modularity, minimal dependencies, and training only the LLM backend, validated across multiple VLMs and visual-math datasets. Empirical findings reveal that RL improves generalization over supervised fine-tuning, with response length and reflection closely intertwined, while seeds strongly affect training dynamics. Overall, Maye provides a reproducible baseline and rich diagnostic toolkit to advance RL-based VLM research and broader adoption.

Abstract

Reinforcement learning (RL) has recently shown strong potential in improving the reasoning capabilities of large language models and is now being actively extended to vision-language models (VLMs). However, existing RL applications in VLMs often rely on heavily engineered frameworks that hinder reproducibility and accessibility, while lacking standardized evaluation protocols, making it difficult to compare results or interpret training dynamics. This work introduces a transparent, from-scratch framework for RL in VLMs, offering a minimal yet functional four-step pipeline validated across multiple models and datasets. In addition, a standardized evaluation scheme is proposed to assess training dynamics and reflective behaviors. Extensive experiments on visual reasoning tasks uncover key empirical findings: response length is sensitive to random seeds, reflection correlates with output length, and RL consistently outperforms supervised fine-tuning (SFT) in generalization, even with high-quality data. These findings, together with the proposed framework, aim to establish a reproducible baseline and support broader engagement in RL-based VLM research.

Figures (8)

• Figure 1: Text-dominant tasks rely on text with visual support; vision-dominant tasks rely on visuals with textual support.
• Figure 2: Overview of Maye framework. The process is divided into four steps. Each step integrates various components, including text and vision data, policy models, and reward signals.
• Figure 3: Overview of evaluation metrics.
• Figure 4: Training set metrics across models and datasets. Red curves show training accuracy (per epoch) and response length (per generation step). Blue curves depict key reflection ratios from \ref{['sec:evaluation_scheme']}, and green curves illustrate the usage trends of the two most frequent and dynamic reflection words per experiment. Shaded regions represent standard deviation across three runs.
• Figure 5: Validation and test accuracy curves across training epochs for different VLMs and datasets. Red lines denote RL, blue lines denote SFT (see \ref{['sec:rl_sft_generalization']}), and green indicate untrained (Vanilla) performance. All curves are averaged over 3 runs, with shaded areas indicating standard deviation.