Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation

TL;DR

Scenethesis addresses the gap in text-to-3D scene generation by fusing language-driven coarse planning with vision-guided spatial refinement and physics-aware optimization. It introduces a training-free agentic workflow that uses an LLM for coarse scene layout, a vision foundation model for image guidance and scene graph extraction, and a Signed Distance Field-based optimization to enforce collision-free, stable object placement, followed by a GPT-4o based judge for spatial coherence. The approach yields higher diversity, layout realism, and physical plausibility than state-of-the-art baselines across indoor and outdoor scenes, demonstrating strong potential for virtual content creation and embodied AI research. Limitations include reliance on retrieval databases for assets and the current inability to synthesize articulated objects, with future work aiming to expand asset banks and enable articulation in 3D generation.

Abstract

Synthesizing interactive 3D scenes from text is essential for gaming, virtual reality, and embodied AI. However, existing methods face several challenges. Learning-based approaches depend on small-scale indoor datasets, limiting the scene diversity and layout complexity. While large language models (LLMs) can leverage diverse text-domain knowledge, they struggle with spatial realism, often producing unnatural object placements that fail to respect common sense. Our key insight is that vision perception can bridge this gap by providing realistic spatial guidance that LLMs lack. To this end, we introduce Scenethesis, a training-free agentic framework that integrates LLM-based scene planning with vision-guided layout refinement. Given a text prompt, Scenethesis first employs an LLM to draft a coarse layout. A vision module then refines it by generating an image guidance and extracting scene structure to capture inter-object relations. Next, an optimization module iteratively enforces accurate pose alignment and physical plausibility, preventing artifacts like object penetration and instability. Finally, a judge module verifies spatial coherence. Comprehensive experiments show that Scenethesis generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.

Figures (17)

• Figure 1: Scenethesis is a framework for text to interactive 3D scene generation. Given a text prompt, Scenethesis leverages both language and visual priors to generate realistic and physical plausible indoor and outdoor environments.
• Figure 2: Unrealistic 3D scenes generated by the LLM-based method (Holodeck yang2024holodeck), exhibiting misplaced objects and oversimplified spatial relations.
• Figure 3: Scenethesis is an agentic framework. The LLM module performs coarse scene planning, estimating rough spatial relationships. The vision module refines this layout by enforcing accurate spatial constraints. The physical-aware optimization iteratively adjusts object placement, ensuring pose alignment and physical plausibility. Finally, a judge module verifies the scene spatial coherence.
• Figure 4: Collision avoidance and stability maintenance.
• Figure 5: Human preference on diverse indoor scenes.