SceneFoundry: Generating Interactive Infinite 3D Worlds

TL;DR

SceneFoundry presents a multi-stage framework for scalable, apartment-scale 3D scene generation conditioned on natural language. It combines an LLM-guided parameter space for floor plans with a diffusion posterior sampling process guided by differentiable constraints that enforce object quantity limits and articulation-aware collision avoidance, followed by a post-hoc walkable-area optimization. The approach introduces four task-specific evaluation metrics to quantify controllability and demonstrates strong performance across conditioned synthesis, layout fidelity, object-count control, functional plausibility, and navigability, outperforming baselines and enabling robust embodied AI training data. This work advances sim-to-real research by delivering semantically coherent, physically usable, and visually diverse 3D interiors suitable for robotic manipulation and navigation tasks.

Abstract

The ability to automatically generate large-scale, interactive, and physically realistic 3D environments is crucial for advancing robotic learning and embodied intelligence. However, existing generative approaches often fail to capture the functional complexity of real-world interiors, particularly those containing articulated objects with movable parts essential for manipulation and navigation. This paper presents SceneFoundry, a language-guided diffusion framework that generates apartment-scale 3D worlds with functionally articulated furniture and semantically diverse layouts for robotic training. From natural language prompts, an LLM module controls floor layout generation, while diffusion-based posterior sampling efficiently populates the scene with articulated assets from large-scale 3D repositories. To ensure physical usability, SceneFoundry employs differentiable guidance functions to regulate object quantity, prevent articulation collisions, and maintain sufficient walkable space for robotic navigation. Extensive experiments demonstrate that our framework generates structurally valid, semantically coherent, and functionally interactive environments across diverse scene types and conditions, enabling scalable embodied AI research.

Figures (10)

• Figure 1: Overview of SceneFoundry. The framework generates apartment-scale 3D scenes from natural language prompts via LLM-guided floor plan generation, diffusion-based placement, and post-optimization ensuring articulated functionality and robot navigability.
• Figure 2: Overview of our apartment-scale generation pipeline. An LLM first guides procedural floor plan generation (Sec. \ref{['sec:llm_guide']}), diffusion posterior guidance generates plausible room bounding boxes (Sec. \ref{['sec:posterior_sampling']}, Sec. \ref{['sec:control_quantity']}, Sec. \ref{['sec:control_articoll']}), and 3D assets from 3D-FRONT/GAPartNet are refined via post-optimization to complete the layout (Sec. \ref{['sec:control_walkable']}).
• Figure 3: Illustration of our LLM-based Guidance. A low penalty (left) produces diverse, non-rectilinear layouts, whereas a high penalty (right) enforces square-shaped room layouts.
• Figure 4: Guidance scheduling during the reverse diffusion process. Object quantity control is applied at $t < 100$ and articulated collision constraint at $t < 1$0, followed by a final walkable-ratio optimization at $t = 0$ to generate a realistic scene.
• Figure 5: Qualitative comparison of conditioned scene synthesis results among PhyScene, ATISS, DiffuScene, and SceneFoundry.