What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom

TL;DR

This work investigates what vision tool-use reinforcement learning (RL) actually learns when equipped with a crop-and-zoom tool. It introduces MED (Measure--Explain--Diagnose) to separate intrinsic capability drift from tool-induced effects, and to decompose tool-induced changes into four terms capturing Gain and Harm from tool calls and tool schemas. Across two backbones with different tool familiarity and six benchmarks, intrinsic drift overwhelmingly drives performance improvements, while tool use primarily reduces harm and shows limited capacity to repair intrinsic failures. The findings suggest current vision tool-use RL tends to coexist with tools rather than master them, motivating future work on more effective tool incorporation and intrinsic capability enhancement. The framework and results provide a mechanistic lens for attributing learning dynamics in multimodal RL systems and could guide the design of more reliable interactive perception agents. $Acc_{ m w}(t)$ and $Acc_{ m wo}(t)$, along with $G(t)$, are central to the attribution, with the key insight that harm reduction often outpaces gains in tool utility.

Abstract

Vision tool-use reinforcement learning (RL) can equip vision-language models with visual operators such as crop-and-zoom and achieves strong performance gains, yet it remains unclear whether these gains are driven by improvements in tool use or evolving intrinsic capabilities.We introduce MED (Measure-Explain-Diagnose), a coarse-to-fine framework that disentangles intrinsic capability changes from tool-induced effects, decomposes the tool-induced performance difference into gain and harm terms, and probes the mechanisms driving their evolution. Across checkpoint-level analyses on two VLMs with different tool priors and six benchmarks, we find that improvements are dominated by intrinsic learning, while tool-use RL mainly reduces tool-induced harm (e.g., fewer call-induced errors and weaker tool schema interference) and yields limited progress in tool-based correction of intrinsic failures. Overall, current vision tool-use RL learns to coexist safely with tools rather than master them.

Figures (8)

• Figure 1: The MED (Measure--Explain--Diagnose) framework for vision tool-use RL.(a) We train a VLM with vision tool-use RL and evaluate each checkpoint under two protocols: tool-free accuracy $Acc_{\mathrm{wo}}$ (intrinsic capability) and tool-available accuracy $Acc_{\mathrm{w}}$. (b) Measure: separate intrinsic drift $f_{\mathrm{wo}}(t)=Acc_{\mathrm{wo}}(t)-Acc_{\mathrm{wo}}(0)$ from tool-induced drift $\Delta_{\mathrm{tool}}(t)$ by tracking the evolution of the gap $G(t)=Acc_{\mathrm{w}}(t)-Acc_{\mathrm{wo}}(t)$. (c) Explain: decompose $G(t)$ into Gains on intrinsic failures $\mathcal{D}_{\mathrm{fail}}(t)$ and Harms on intrinsic successes $\mathcal{D}_{\mathrm{succ}}(t)$, further distinguishing tool-call effects from schema-only effects (four terms to the right of the equal sign). (d) Diagnose: probe the underlying mechanisms behind each term's evolution to identify what changes in tool use drive gains or harms over training.
• Figure 2: Quantifying Intrinsic and Tool-Induced Drift. We aggregate learning dynamics across six benchmarks (VStar, HR-Bench 4k/8k, VisualProbe Easy/Medium/Hard), evaluated every 80 gradient steps (21 checkpoints). Left: Tool-free and tool-available drifts. We normalize per-benchmark $\Delta$ accuracy to $[-1,1]$ and then average across benchmarks to compute curves. The grey area ($|B_{\mathrm{wo}}|$) quantifies the cumulative magnitude of intrinsic drift ($f_\mathrm{wo}$). The colored area represents the magnitude of tool-induced drift ($\Delta_{\mathrm{tool}}$), Green indicates positive relative gain ($f_w > f_{wo}$), while red indicates negative relative drift ($f_w < f_{wo}$). Color intensity corresponds to the tool call rate. The top progress bar displays the tool contribution ratio ($S_{tool}$), i.e., the proportion of total drift magnitude attributed to tool effects. Right: Absolute accuracy ($Acc_w$ and $Acc_{wo}$) is averaged directly across all benchmarks. All curves are smoothed for visualization only. Full details on area calculation, normalization, aggregation and smoothing are provided in §\ref{['app:normalize_drift_agg']}.
• Figure 3: Decomposition of Tool-Induced Performance Gap $G(t)$. Averaged across six benchmarks. \ref{['eq:decomposition']} breaks down the net gap $G(t)$ (yellow diamonds) into Gross Gain (green; T1+T2) and Gross Harm (red; T3+T4). Gross Gain consists of Call Gain (T1; intrinsic failures corrected via tool execution) and Schema Gain (T2; schema-only recovery without tool calls). Gross Harm consists of Call Harm (T3; intrinsic successes flipped to errors after tool calls) and Schema Harm (errors induced by the tool schema without calls). (a) Qwen2.5-VL: Call Gain (T1) quickly reverses the initial negative gap, then plateaus and slightly declines. (b) Qwen3-VL: Gross Gain shrinks mainly due to declining Call Gain. Both models show concurrent reductions in Gross Gain and Gross Harm at later stages. Overall, the dynamics are consistent with harm reduction (suppressed detrimental usage) rather than continued gain maximization.
• Figure 4: Factor Decomposition of Tool-Induced Effects. We show the temporal evolution of the four terms, factorized into Mass, Policy, and Quality, for (a) Qwen2.5-VL-Instruct and (b) Qwen3-VL-Instruct. In each subplot, the thick line shows the term value (left axis), and thin lines show its factors (right axis): Mass (grey, $P(\mathcal{D})$), Policy (blue, $P(a\mid\mathcal{D})$), and Quality (orange, $P(o\mid a,\mathcal{D})$).
• Figure 5: Robustness to the Moving Failure Set. The Call-Gain quality $P(\checkmark \mid c, \mathcal{D}_{\mathrm{fail}})$ evaluated under different failure-set definitions: the current failure set $\mathcal{D}_{\mathrm{fail}}(t)$ (Dynamic), the fixed initial cohort $\mathcal{D}_{\mathrm{fail}}(0)$ (Fixed), and persistent failures $\mathcal{D}_{\mathrm{fail}}(0)\cap \mathcal{D}_{\mathrm{fail}}(t)$. Improvement is observed on the fixed cohort but remains limited on the current and persistent failure sets.