Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning

TL;DR

The paper tackles the challenge of turning vision into editable graphics programs by addressing the grounding gaps in one-shot vision-language models. It introduces VIGA, an execution-grounded agent that iteratively writes and executes scene programs, verifi es renders from multiple viewpoints, and updates a memory of plans and diffs in a write-run-render-compare-revise loop. Key contributions include the memory-augmented, task-agnostic framework with a versatile skill library, the BlenderBench benchmark for interleaved multimodal reasoning, and substantial empirical gains across 2D, 3D, and 4D tasks without model fine-tuning. The approach enables robust inverse-graphics reasoning and offers a rigorous testbed for evaluating foundation models in multimodal, programmable visual tasks with practical implications for 3D editing, reconstruction, and robotic simulation.

Abstract

Vision-as-inverse-graphics, the concept of reconstructing an image as an editable graphics program is a long-standing goal of computer vision. Yet even strong VLMs aren't able to achieve this in one-shot as they lack fine-grained spatial and physical grounding capability. Our key insight is that closing this gap requires interleaved multimodal reasoning through iterative execution and verification. Stemming from this, we present VIGA (Vision-as-Inverse-Graphic Agent) that starts from an empty world and reconstructs or edits scenes through a closed-loop write-run-render-compare-revise procedure. To support long-horizon reasoning, VIGA combines (i) a skill library that alternates generator and verifier roles and (ii) an evolving context memory that contains plans, code diffs, and render history. VIGA is task-agnostic as it doesn't require auxiliary modules, covering a wide range of tasks such as 3D reconstruction, multi-step scene editing, 4D physical interaction, and 2D document editing, etc. Empirically, we found VIGA substantially improves one-shot baselines on BlenderGym (35.32%) and SlideBench (117.17%). Moreover, VIGA is also model-agnostic as it doesn't require finetuning, enabling a unified protocol to evaluate heterogeneous foundation VLMs. To better support this protocol, we introduce BlenderBench, a challenging benchmark that stress-tests interleaved multimodal reasoning with graphics engine, where VIGA improves by 124.70%.

Figures (7)

• Figure 1: VIGA crafts a 3D graphics scene from a single image through iteration. Given a single 2D image (e.g. a bathroom photo or a stylized drawing of the Golden Gate Bridge), VIGA alternates between a generation step where it produces and executes code to render a scene, and a verification step, where it inspects the rendered scene, identifies visual issues, and feeds them back to guide the next edit. Through these iterations, VIGA progressively refines layout, geometry, and lighting, eventually reconstructing a coherent 3D scene.
• Figure 2: The main pipeline of VIGA. During generation, VIGA consumes the full memory trajectory---including past code versions and rendered scenes---and produces the next scene program, which is then executed to generate a new render. During verification, VIGA invokes its verification tools to adjust camera viewpoints, thoroughly inspect the current scene, identify the dominant discrepancy, and feed this feedback into the next generation step. We illustrate the scene editing setting here. VIGA can also synthesize scenes from scratch given a goal specification (as in \ref{['fig:teaser_multi_agent']}).
• Figure 3: 3D scene results. We show the input on the left, and the final output graphics scene from three viewpoints on the right.
• Figure 4: 4D scene results. VIGA constructs a complete 4D scene from a single input image and a text instruction, producing motions that reflect real-world physics.
• Figure 5: Agent trajectories across different tasks. Examples from BlenderGym and SlideBench. At each verification step, the agent identifies a different visual issue in the rendered scene and then refines the code accordingly in the next iteration.