Transductive Visual Programming: Evolving Tool Libraries from Experience for Spatial Reasoning

TL;DR

The paper tackles the challenge of precise 3D spatial reasoning in visual scenes by introducing Transductive Visual Programming (TVP), a closed-loop framework that learns tools from solved experiences. TVP maintains an Example Library of concrete solutions and a Tool Library of abstracted functions, iteratively solving problems, extracting patterns, and expanding its toolkit. Empirical results on Omni3D-Bench show state-of-the-art performance and improved program efficiency, with tools transferring to unseen spatial tasks, indicating robust generalization. This experience-driven, transductive approach provides a scalable paradigm for self-evolving visual-programming agents in complex spatial domains.

Abstract

Spatial reasoning in 3D scenes requires precise geometric calculations that challenge vision-language models. Visual programming addresses this by decomposing problems into steps calling specialized tools, yet existing methods rely on either fixed toolsets or speculative tool induction before solving problems, resulting in suboptimal programs and poor utilization of induced tools. We present Transductive Visual Programming (TVP), a novel framework that builds new tools from its own experience rather than speculation. TVP first solves problems using basic tools while accumulating experiential solutions into an Example Library, then abstracts recurring patterns from these programs into reusable higher-level tools for an evolving Tool Library. This allows TVP to tackle new problems with increasingly powerful tools learned from experience. On Omni3D-Bench, TVP achieves state-of-the-art performance, outperforming GPT-4o by 22% and the previous best visual programming system by 11%. Our transductively learned tools are used 5x more frequently as core program dependency than inductively created ones, demonstrating more effective tool discovery and reuse. The evolved tools also show strong generalization to unseen spatial tasks, achieving superior performance on benchmarks from SpatialScore-Hard collection without any testset-specific modification. Our work establishes experience-driven transductive tool creation as a powerful paradigm for building self-evolving visual programming agents that effectively tackle challenging spatial reasoning tasks. We release our code at https://transductive-visualprogram.github.io/.

Figures (15)

• Figure 1: (a) Prior methods operate in an open-loop manner: tools are created without experience from solving problems. (b) TVP maintains both an Example Library of experiential solutions and a Tool Library of abstracted functions, forming a closed-loop system where tools are created from proven solution experience, and are then reused to guide future problem-solving.
• Figure 2: Tool usage distribution: transductive (TVP) vs. inductive (VADAR) abstraction.
• Figure 2: Results on benchmarks from sampled SpatialScore-Hard collection. TVP generalizes zero-shot with only libraries built from Omni3D-Bench. Best bold, second underlined.
• Figure 3: TVP's dual-library architecture. (Phase I) Problem-solving and experience accumulation: For each query, TVP retrieves similar examples from the Example Library and generates programs using the current Tool Library; high-quality solutions join the Example Library. (Phase II) Tool abstraction: Accumulated examples are clustered, and common patterns are abstracted into new tools, which, if passed validation, are added to the Tool Library for future use.
• Figure 4: Tool Library maintenance via merging (\ref{['sec:method_maintenance']}). Functionally similar tools emerge from different clusters; TVP merges them into a more general abstraction covering both use cases.