Imaginarium: Vision-guided High-Quality 3D Scene Layout Generation

TL;DR

Imaginarium introduces a vision-guided pipeline that converts 2D image guidance into high-quality 3D scene layouts using a large, curated asset library and a diffusion-based prompt-to-image expander. The method integrates semantic/geometry parsing, scene-graph reasoning, and constrained optimization to retrieve assets and estimate rotations, translations, and scales, followed by refinement under physical and spatial constraints. Extensive human studies and quantitative metrics demonstrate superior layout richness, realism, and fidelity compared with strong baselines, and the work provides an open-source dataset of assets and layouts to benefit the research community. The approach offers a practical and scalable pathway to generate diverse, coherent 3D scenes from textual prompts or guide images, reducing manual artistry and enabling rapid content creation.

Abstract

Generating artistic and coherent 3D scene layouts is crucial in digital content creation. Traditional optimization-based methods are often constrained by cumbersome manual rules, while deep generative models face challenges in producing content with richness and diversity. Furthermore, approaches that utilize large language models frequently lack robustness and fail to accurately capture complex spatial relationships. To address these challenges, this paper presents a novel vision-guided 3D layout generation system. We first construct a high-quality asset library containing 2,037 scene assets and 147 3D scene layouts. Subsequently, we employ an image generation model to expand prompt representations into images, fine-tuning it to align with our asset library. We then develop a robust image parsing module to recover the 3D layout of scenes based on visual semantics and geometric information. Finally, we optimize the scene layout using scene graphs and overall visual semantics to ensure logical coherence and alignment with the images. Extensive user testing demonstrates that our algorithm significantly outperforms existing methods in terms of layout richness and quality. The code and dataset will be available at https://github.com/HiHiAllen/Imaginarium.

Figures (18)

• Figure 1: Overview of our method. We first transforms a text prompt into a detailed 2D guide image using a fine-tuned model, ensuring stylistic consistency with our asset library. This image is then analyzed for semantic, geometric, and relational information, guiding the retrieval, transformation estimation, and optimization of 3D assets into the final, coherent layout. See Appendix \ref{['subsubsec:supp_vis']} for additional visualizations of intermediate steps.
• Figure 2: Overview of our high-quality 3D scene layout dataset: (a) Representative 3D scenes with interior layouts. (b) Diverse 3D assets from our collection. (c) Structured metadata schema for scenes and assets. (d) Comparison with 3D-Future, highlighting our dataset's superior variety and complexity.
• Figure 3: (a) Scene graph constraints extracted by our algorithm. (b) Close-up of the support relationship tree structure (highlighted in red box in (a)).
• Figure 4: Coarse-to-fine view selection. Coarse selection ranks views by keypoint match quality, while fine selection uses homography transformation to identify the most viewpoint-similar match (selecting $v_{1}^{\text{vis}}$ in this example).
• Figure 5: Top-view illustration of candidates' geometric enhancement. Each row compares orientation estimations for different query scenarios, showing ground truth (GT), OBB-based ($v_{\ast}^{\text{obb}}$), and vision-based ($v_1^{\text{vis}}$) estimations. The best estimation ($v_{\text{best}}$) is highlighted, demonstrating the adaptive integration of geometric guidance.