WorldAgents: Can Foundation Image Models be Agents for 3D World Models?

Abstract

Given the remarkable ability of 2D foundation image models to generate high-fidelity outputs, we investigate a fundamental question: do 2D foundation image models inherently possess 3D world model capabilities? To answer this, we systematically evaluate multiple state-of-the-art image generation models and Vision-Language Models (VLMs) on the task of 3D world synthesis. To harness and benchmark their potential implicit 3D capability, we propose an agentic framing to facilitate 3D world generation. Our approach employs a multi-agent architecture: a VLM-based director that formulates prompts to guide image synthesis, a generator that synthesizes new image views, and a VLM-backed two-step verifier that evaluates and selectively curates generated frames from both 2D image and 3D reconstruction space. Crucially, we demonstrate that our agentic approach provides coherent and robust 3D reconstruction, producing output scenes that can be explored by rendering novel views. Through extensive experiments across various foundation models, we demonstrate that 2D models do indeed encapsulate a grasp of 3D worlds. By exploiting this understanding, our method successfully synthesizes expansive, realistic, and 3D-consistent worlds.

Figures (7)

• Figure 1: WorldAgents employs 2D foundation models as agents in an iterative process to extract realistic, coherent 3D scenes from their learned distributions. The scene generation process is orchestrated by a VLM agent (director) that generates prompts for new views to synthesize, fulfilled by an image generation model that creates these views, and verified by a VLM agent that assesses both 2D and 3D consistency of the new synthesized image(s). The output is a sci-fi lab. scene reconstructed with 3DGS, and can be visualized from novel views for exploration.
• Figure 2: Method overview. Our method employs a multi-agent approach comprising a Director, a Generator, and a Verifier to construct coherent 3D scenes using image diffusion models. The Director guides the overall process by formulating novel prompts. The Generator then leverages sequential inpainting to synthesize 3D-consistent views. Subsequently, the Verifier evaluates these generated views to ensure rigorous multi-view consistency. Finally, the verified frames are reconstructed into a 3D Gaussian Splatting (3DGS) representation.
• Figure 3: Visual results from WorldAgents. Our method can generate diverse scenes that are populated with various objects in a clean and coherent way by following the text prompt.
• Figure 4: Qualitative comparison of different image models and VLMs. The image models that we experimented with all showed satisfactory 3D scene generation results. However, there are subtle differences between them aligning with the complexity of the individual model.
• Figure 5: Qualitative baseline comparison. Our method generates visually appealing results with multiple objects placed in the scene nicely, without having artifacts or objectless regions unlike baselines.