Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement

Zhengfei Kuang, Rui Lin, Long Zhao, Gordon Wetzstein, Saining Xie, Sanghyun Woo

TL;DR

VULCAN introduces a tool-augmented, multi-agent framework for iterative 3D object arrangement, addressing the shortcomings of single-step MLLM edits by decoupling planning, execution, and evaluation. It combines MCP-based visual tools with a constraint-based collision-free solver and an adaptive backtracking search to reliably generate physically plausible, non-colliding sequences that satisfy user instructions. The approach is validated on a 25-scene benchmark with 111 tasks, outperforming baselines in collisions, floating artifacts, plausibility, and instruction-consistency, with human evaluators also preferring the results. The work advances long-horizon, 3D scene manipulation by integrating grounded perception, robust planning, and reliable execution, while acknowledging current single-view limitations and potential multi-view extensions as MLLMs evolve.

Abstract

Despite the remarkable progress of Multimodal Large Language Models (MLLMs) in 2D vision-language tasks, their application to complex 3D scene manipulation remains underexplored. In this paper, we bridge this critical gap by tackling three key challenges in 3D object arrangement task using MLLMs. First, to address the weak visual grounding of MLLMs, which struggle to link programmatic edits with precise 3D outcomes, we introduce an MCP-based API. This shifts the interaction from brittle raw code manipulation to more robust, function-level updates. Second, we augment the MLLM's 3D scene understanding with a suite of specialized visual tools to analyze scene state, gather spatial information, and validate action outcomes. This perceptual feedback loop is critical for closing the gap between language-based updates and precise 3D-aware manipulation. Third, to manage the iterative, error-prone updates, we propose a collaborative multi-agent framework with designated roles for planning, execution, and verification. This decomposition allows the system to robustly handle multi-step instructions and recover from intermediate errors. We demonstrate the effectiveness of our approach on a diverse set of 25 complex object arrangement tasks, where it significantly outperforms existing baselines. Website: vulcan-3d.github.io

VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement

TL;DR

VULCAN introduces a tool-augmented, multi-agent framework for iterative 3D object arrangement, addressing the shortcomings of single-step MLLM edits by decoupling planning, execution, and evaluation. It combines MCP-based visual tools with a constraint-based collision-free solver and an adaptive backtracking search to reliably generate physically plausible, non-colliding sequences that satisfy user instructions. The approach is validated on a 25-scene benchmark with 111 tasks, outperforming baselines in collisions, floating artifacts, plausibility, and instruction-consistency, with human evaluators also preferring the results. The work advances long-horizon, 3D scene manipulation by integrating grounded perception, robust planning, and reliable execution, while acknowledging current single-view limitations and potential multi-view extensions as MLLMs evolve.

Abstract

Despite the remarkable progress of Multimodal Large Language Models (MLLMs) in 2D vision-language tasks, their application to complex 3D scene manipulation remains underexplored. In this paper, we bridge this critical gap by tackling three key challenges in 3D object arrangement task using MLLMs. First, to address the weak visual grounding of MLLMs, which struggle to link programmatic edits with precise 3D outcomes, we introduce an MCP-based API. This shifts the interaction from brittle raw code manipulation to more robust, function-level updates. Second, we augment the MLLM's 3D scene understanding with a suite of specialized visual tools to analyze scene state, gather spatial information, and validate action outcomes. This perceptual feedback loop is critical for closing the gap between language-based updates and precise 3D-aware manipulation. Third, to manage the iterative, error-prone updates, we propose a collaborative multi-agent framework with designated roles for planning, execution, and verification. This decomposition allows the system to robustly handle multi-step instructions and recover from intermediate errors. We demonstrate the effectiveness of our approach on a diverse set of 25 complex object arrangement tasks, where it significantly outperforms existing baselines. Website: vulcan-3d.github.io
Paper Structure (38 sections, 5 equations, 23 figures, 4 tables, 2 algorithms)

This paper contains 38 sections, 5 equations, 23 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. 3D Object Manipulation and Arrangement
  4. MLLMs for 3D
  5. Problem Formulation
  6. Multi-Agent Framework
  7. Planning with Global Context
  8. Tool-Assisted Action Execution
  9. Evaluation and Filtering
  10. Visual Annotation
  11. Adaptive Backtracking
  12. Multi-Tool Library
  13. Visual Probing
  14. Constraint Formulation
  15. Optimization
  16. ...and 23 more sections

Figures (23)

  • Figure 1: VULCAN plans and executes multiple actions for complex object arrangement tasks given input image and user prompt.
  • Figure 2: VULCAN Overview. (a) Our method solves a long-horizon task through an iterative multi-agent process. (b) The Planner agent examines the global context (the user instruction and all previous rendered states) to formulate a concrete plan for the current movement. (c) The Executor implements this single-step plan in the 3D scene using API tools and solvers. (d) A set of Evaluators and an automatic floating check assess the execution quality. The entire "Plan-Execute-Evaluate" loop repeats until the Planner validates that the final arrangement fulfills the original user instruction.
  • Figure 3: Impact of visual annotation. (a) Edit visualization improves the MLLM's spatial arrangement recognition; (b) Coordinate annotations enhance its object localization accuracy.
  • Figure 4: Visual Probing Tools. For each API, we present an example function call format, along with a result visualization.
  • Figure 5: Qualitative Comparisons. We compare results using two types of guidance: detailed per-step instructions (Top) and general, abstract instructions (Bottom). Top: When given per-step instructions, our method outperforms the baselines. Bottom: Using general instructions, our multi-step approach produces reasonable intermediate states and yields a significantly better final state (marked in red) compared to single-step baselines.
  • ...and 18 more figures