Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning

TL;DR

This paper presents VisualToolBench, the first benchmark explicitly targeting think-with-images for multimodal LLMs, where models must perceive, transform, and reason with images using a variety of tools. Comprising 1,204 open-ended tasks across five domains, the benchmark employs rubric-based, multi-dimensional evaluation and a standardized tool API (including a vision-centric Python image-processing tool) to capture complex, multi-step workflows. Across 16 representative MLLMs, results show substantial gaps: the best model (GPT-5-think) achieves only 18.68% pass rate, and most models underperform, underscoring the need for improved tool integration and active visual manipulation. The study also reveals that proactive tool use and broader image-editing behavior correlate with higher performance, while perceptual errors remain the primary bottleneck, offering concrete directions for advancing robust visual intelligence in MLLMs.

Abstract

Multimodal Large Language Models (MLLMs) are increasingly applied in real-world scenarios where user-provided images are often imperfect, requiring active image manipulations such as cropping, editing, or enhancement to uncover salient visual cues. Beyond static visual perception, MLLMs must also think with images: dynamically transforming visual content and integrating it with other tools to solve complex tasks. However, this shift from treating vision as passive context to a manipulable cognitive workspace remains underexplored. Most existing benchmarks still follow a think about images paradigm, where images are regarded as static inputs. To address this gap, we introduce VisualToolBench, a visual tool-use reasoning benchmark that rigorously evaluates MLLMs' ability to perceive, transform, and reason across complex visual-textual tasks under the think-with-images paradigm. VisualToolBench comprises 1,204 challenging, open-ended vision tasks (603 single-turn, 601 multi-turn) spanning across five diverse domains, each paired with detailed rubrics to enable systematic evaluation. Our evaluation shows that current MLLMs struggle with tasks requiring effective integration of vision and general-purpose tools. Even the strongest model (GPT-5-think) reaches only 18.68% pass rate. We further observe divergent tool-use behaviors, with OpenAI models benefiting from diverse image manipulations while Gemini-2.5-pro shows no improvement. By introducing the first benchmark centered on think with images, VisualToolBench offers critical insights for advancing visual intelligence in MLLMs.

Figures (11)

• Figure 1: Topic distribution.
• Figure 2: Statistics of VisualToolBench.
• Figure 2: Demonstration example from VisualToolBench (single-turn, generalist domain, region switch Q&A). The key visual content needed to solve the task is distributed across different regions of the image, requiring the model to crop multiple regions of interest (RoIs) for accurate perception and reasoning. Each task is paired with a detailed set of rubrics to evaluate model's responses. From these rubrics, we derive both a weighted rubric score between 0 and 1 and a binary pass/fail outcome, depending on whether critical rubrics are satisfied.
• Figure 3: APR across task categories for the top five models on VisualToolBench. Bars report model performance on each category, with exact APR values labeled above.
• Figure 4: Relationship between APR and tool-use behaviors across models. Left: APR versus tool-use proactivity. Right: APR versus average tool call volume per task.