Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs

TL;DR

The paper addresses the challenge of enabling robust tool-integrated reasoning in vision-language models by introducing VISTA-Gym, a scalable training environment that combines 7 reasoning tasks from 13 datasets with a unified 26-tool interface. It presents VISTA-R1, a VLM-based agent trained in two stages (behavioral cloning followed by online RL with GRPO) and guided by a multi-component reward design that enforces a think→tool_call→answer protocol. Empirical results across 11 benchmarks show that VISTA-R1‑8B outperforms comparable baselines by 9.51%–18.72% when tools are used, and demonstrates strong generalization to out-of-distribution tasks, with notable parameter efficiency. The work provides a scalable, extensible platform for advancing thinking-with-images capabilities in open-source VLMs, with practical implications for multi-step visual problem solving and tool coordination in AI systems.

Abstract

While recent vision-language models (VLMs) demonstrate strong image understanding, their ability to "think with images", i.e., to reason through multi-step visual interactions, remains limited. We introduce VISTA-Gym, a scalable training environment for incentivizing tool-integrated visual reasoning capabilities in VLMs. VISTA-Gym unifies diverse real-world multimodal reasoning tasks (7 tasks from 13 datasets in total) with a standardized interface for visual tools (e.g., grounding, parsing), executable interaction loops, verifiable feedback signals, and efficient trajectory logging, enabling visual agentic reinforcement learning at scale. While recent VLMs exhibit strong text-only reasoning, both proprietary and open-source models still struggle with tool selection, invocation, and coordination. With VISTA-Gym, we train VISTA-R1 to interleave tool-use with agentic reasoning via multi-turn trajectory sampling and end-to-end reinforcement learning. Extensive experiments across 11 public reasoning-intensive VQA benchmarks show that VISTA-R1-8B outperforms state-of-the-art baselines with similar sizes by 9.51%-18.72%, demonstrating VISTA-Gym as an effective training ground to unlock the tool-integrated reasoning capabilities for VLMs.

Figures (9)

• Figure 1: Directly augmenting VLMs with tools significantly degrades accuracy (w/ T), yet intrinsic reasoning offers limited gains on complex VQA (w/ R). Supplying tool‑selection prior knowledge and interleaving reasoning with tool execution improve performance (w/ T&R); gains are task‑dependent for commercial VLMs, while small open‑source VLMs remain particularly struggling.
• Figure 2: Overview of VISTA-Gym. VISTA-Gym contains a comprehensive suite of reasoning-intensive VQA tasks and tools in an interactive execution environment, scaling visual-centric tool-integrated agentic training for VLM agents.
• Figure 3: Top tool call distribution of different tasks in SFT data.
• Figure 4: Ablation studies and diversity analysis with InternVL3-8B as backbone VLM.
• Figure 5: Quality of expert thinking trajectories by length.