PyVision: Agentic Vision with Dynamic Tooling

TL;DR

PyVision introduces an agentic, multi-turn framework that lets multimodal LLMs dynamically generate and execute Python-based tools to solve visual reasoning tasks. By replacing static toolchains with on-the-fly, task-tailored tooling and a robust runtime loop, the approach yields consistent, model-dependent performance gains across diverse benchmarks. The work provides a taxonomy of generated tools and analyzes how tool usage varies with domain and task, demonstrating that dynamic tooling amplifies the strengths of the backend model. These findings suggest a shift toward genuinely autonomous visual reasoning where models invent and refine computational aids as needed. Practically, PyVision improves capability in perception-heavy and abstract-reasoning tasks and highlights the importance of cross-turn code generation and safe runtime execution for scalable multimodal reasoning.

Abstract

LLMs are increasingly deployed as agents, systems capable of planning, reasoning, and dynamically calling external tools. However, in visual reasoning, prior approaches largely remain limited by predefined workflows and static toolsets. In this report, we present PyVision, an interactive, multi-turn framework that enables MLLMs to autonomously generate, execute, and refine Python-based tools tailored to the task at hand, unlocking flexible and interpretable problem-solving. We develop a taxonomy of the tools created by PyVision and analyze their usage across a diverse set of benchmarks. Quantitatively, PyVision achieves consistent performance gains, boosting GPT-4.1 by +7.8% on V* and Claude-4.0-Sonnet by +31.1% on VLMsAreBlind-mini. These results point to a broader shift: dynamic tooling allows models not just to use tools, but to invent them, advancing toward more agentic visual reasoning.

Figures (10)

• Figure 1: Overcoming Illusory Heuristics with Code. This visual puzzle mimics the well-known Ebbinghaus illusion jaeger2015ebbinghaus, but with a twist: it reverses the typical size context, making the correct answer visually obvious to humans. Yet, a standard MLLM gpt4.1 mistakenly recalls the well-documented illusion template to answer "same size". In contrast, PyVision behaves agentically, probing pixel values, segmenting objects, and computing the actual sizes via on-the-fly Python code to reach the correct answer. This example highlights how dynamic tooling enables adaptive, grounded, verifiable visual reasoning beyond superficial pattern matching.
• Figure 2: PyVision, an interactive and multi-turn framework capable of dynamic tool generation, designed for multimodal reasoning. In an inference session, PyVision performs n+1 interaction turns with the Python interpreter. In the figure, code_block_i refers to the generated Python code by the MLLM in the i-th turn, and mm_clue_i the executed multi-modal outputs by the Python interpreter. This loop continues until the MLLM outputs a final answer.
• Figure 3: Case Study: Visual Search requires language-based world knowledge, vision-based contextual understanding, and iterative refinement to precisely locate specific visual elements. In this case from V* wu2024vstar, the agent must identify text on a small advertising board in a complex outdoor scene. The target occupies a minimal image area, requiring adaptive cropping. This showcases how dynamic tool generation enables flexible spatial reasoning. The agent generates custom Python codes across three iterative turns, refining coordinate-based cropping tools based on visual feedback, and eventually isolating the advertising board required by the user query. The agent then performs OCR and extracts key text such as "YOGA" and correctly identifies the business as a yoga studio.
• Figure 4: Case Study: Medical Imaging Analysis often requires high visual sensitivity and domain-specific reasoning to solve subtle medical problems. This case from OmniMedVQA hu2024omnimedvqa involves identifying a specific abnormality in a fundus image of the retina. PyVision is tasked with diagnosing the image without prior domain-specific modules. To begin, PyVision generates a visualization tool that applies histogram equalization to enhance contrast in key regions, a standard technique in medical imaging analysis. It then creates a second tool to plot the intensity histogram, confirming the absence of abnormal peaks. Within a few iterations, PyVision constructs a custom diagnostic pipeline from scratch, correctly concluding that no specific abnormalities are present. This highlights PyVision’s ability to generate interpretable, domain-adapted tools for complex medical tasks.
• Figure 5: Case Study: Symbolic Visual Puzzles require precise spatial reasoning and the recognition of overlapping or closely positioned geometric primitives, a task that is simple for humans, yet surprisingly difficult for powerful MLLMs when asked directly vlmsareblind. In contrast, the regular structure of such puzzles makes them well-suited to code-based solutions. This is a nested squares counting task from VLMsAreBlind vlmsareblind. First, PyVision applies edge detection using skimage.measure.find_contours and identifies ten contours. Then, it infers that the number of contours corresponds to five nested squares, as each square contributes an inner and outer edge. To validate this, the model performs numerical analysis and prints the sorted perimeters of the detected contours. In the third stage, it performs a double-check and confidently confirms the correct answer: five nested squares.