G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning

TL;DR

The paper tackles the challenge of translating vision-language models into effective decision-making within visually rich, interactive environments. It introduces VLM-Gym, a scalable RL platform with four visual games, and two model series, G0 (RL-only) and G1 (perception-enhanced cold start with distillation), to study how perception and reasoning co-evolve during training. G0 demonstrates emergent perception and reasoning that surpasses several strong baselines, while G1 achieves even stronger performance, outperforming the teacher and Claude-3.7-Sonnet-Thinking. A key finding is that perception and reasoning mutually bootstrap each other during RL, suggesting general strategies for advancing VLMs as capable interactive agents. The work provides open-source tools and insights that could significantly impact multimodal RL research and practical autonomous agents in visually complex domains.

Abstract

Vision-Language Models (VLMs) excel in many direct multimodal tasks but struggle to translate this prowess into effective decision-making within interactive, visually rich environments like games. This ``knowing-doing'' gap significantly limits their potential as autonomous agents, as leading VLMs often performing badly in simple games. To address this, we introduce VLM-Gym, a curated reinforcement learning (RL) environment featuring diverse visual games with unified interfaces and adjustable, compositional difficulty, specifically designed for scalable multi-game parallel training. Leveraging VLM-Gym, we train G0 models using pure RL-driven self-evolution, which demonstrate emergent perception and reasoning patterns. To further mitigate challenges arising from game diversity, we develop G1 models. G1 incorporates a perception-enhanced cold start prior to RL fine-tuning. Our resulting G1 models consistently surpass their teacher across all games and outperform leading proprietary models like Claude-3.7-Sonnet-Thinking. Systematic analysis reveals an intriguing finding: perception and reasoning abilities mutually bootstrap each other throughout the RL training process. Source code including VLM-Gym and RL training are released at https://github.com/chenllliang/G1 to foster future research in advancing VLMs as capable interactive agents.

Figures (14)

• Figure 1: Comparison of different models on games from VLM-Gym.
• Figure 2: Key features of VLM-Gym. We illustrate them using the Shisen-Sho game as an example.
• Figure 3: Average game reward curves of different games for G0 models during RL process.
• Figure 4: The explored perception and reasoning patterns during G0 RL training in Shisen-Sho Game.
• Figure 5: Localization patterns count during G0 RL training for different games.