Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions

Haiyang Zhou, Xinhua Cheng, Wangbo Yu, Yonghong Tian, Li Yuan

TL;DR

HoloDreamer tackles the challenge of generating view-consistent, fully enclosed 3D scenes from text by first producing a high-quality stylized 360-degree panorama and then reconstructing the 3D scene with fast 3D-Gaussian Splatting. The Stylized Equirectangular Panorama Generation ensures geometry-faithful, stylistically rich panoramas, while the Enhanced Two-Stage Panorama Reconstruction introduces multi-view supervision and inpainting-driven refinement to yield robust, watertight 3D reconstructions. Extensive experiments against baselines show superior visual harmony, reconstruction quality, and rendering robustness for fully enclosed scenes. The approach demonstrates a practical, scalable route for text-driven 3D content across indoor and outdoor environments with broad applications in VR, gaming, and film.

Abstract

3D scene generation is in high demand across various domains, including virtual reality, gaming, and the film industry. Owing to the powerful generative capabilities of text-to-image diffusion models that provide reliable priors, the creation of 3D scenes using only text prompts has become viable, thereby significantly advancing researches in text-driven 3D scene generation. In order to obtain multiple-view supervision from 2D diffusion models, prevailing methods typically employ the diffusion model to generate an initial local image, followed by iteratively outpainting the local image using diffusion models to gradually generate scenes. Nevertheless, these outpainting-based approaches prone to produce global inconsistent scene generation results without high degree of completeness, restricting their broader applications. To tackle these problems, we introduce HoloDreamer, a framework that first generates high-definition panorama as a holistic initialization of the full 3D scene, then leverage 3D Gaussian Splatting (3D-GS) to quickly reconstruct the 3D scene, thereby facilitating the creation of view-consistent and fully enclosed 3D scenes. Specifically, we propose Stylized Equirectangular Panorama Generation, a pipeline that combines multiple diffusion models to enable stylized and detailed equirectangular panorama generation from complex text prompts. Subsequently, Enhanced Two-Stage Panorama Reconstruction is introduced, conducting a two-stage optimization of 3D-GS to inpaint the missing region and enhance the integrity of the scene. Comprehensive experiments demonstrated that our method outperforms prior works in terms of overall visual consistency and harmony as well as reconstruction quality and rendering robustness when generating fully enclosed scenes.

HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions

TL;DR

HoloDreamer tackles the challenge of generating view-consistent, fully enclosed 3D scenes from text by first producing a high-quality stylized 360-degree panorama and then reconstructing the 3D scene with fast 3D-Gaussian Splatting. The Stylized Equirectangular Panorama Generation ensures geometry-faithful, stylistically rich panoramas, while the Enhanced Two-Stage Panorama Reconstruction introduces multi-view supervision and inpainting-driven refinement to yield robust, watertight 3D reconstructions. Extensive experiments against baselines show superior visual harmony, reconstruction quality, and rendering robustness for fully enclosed scenes. The approach demonstrates a practical, scalable route for text-driven 3D content across indoor and outdoor environments with broad applications in VR, gaming, and film.

Abstract

3D scene generation is in high demand across various domains, including virtual reality, gaming, and the film industry. Owing to the powerful generative capabilities of text-to-image diffusion models that provide reliable priors, the creation of 3D scenes using only text prompts has become viable, thereby significantly advancing researches in text-driven 3D scene generation. In order to obtain multiple-view supervision from 2D diffusion models, prevailing methods typically employ the diffusion model to generate an initial local image, followed by iteratively outpainting the local image using diffusion models to gradually generate scenes. Nevertheless, these outpainting-based approaches prone to produce global inconsistent scene generation results without high degree of completeness, restricting their broader applications. To tackle these problems, we introduce HoloDreamer, a framework that first generates high-definition panorama as a holistic initialization of the full 3D scene, then leverage 3D Gaussian Splatting (3D-GS) to quickly reconstruct the 3D scene, thereby facilitating the creation of view-consistent and fully enclosed 3D scenes. Specifically, we propose Stylized Equirectangular Panorama Generation, a pipeline that combines multiple diffusion models to enable stylized and detailed equirectangular panorama generation from complex text prompts. Subsequently, Enhanced Two-Stage Panorama Reconstruction is introduced, conducting a two-stage optimization of 3D-GS to inpaint the missing region and enhance the integrity of the scene. Comprehensive experiments demonstrated that our method outperforms prior works in terms of overall visual consistency and harmony as well as reconstruction quality and rendering robustness when generating fully enclosed scenes.
Paper Structure (13 sections, 4 equations, 10 figures, 4 tables)

This paper contains 13 sections, 4 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. 3D Representation
  4. 3D Scene Generation
  5. Panorama Generation
  6. Method
  7. Stylized Equirectangular Panorama Generation
  8. Enhanced Two-Stage Panorama Reconstruction
  9. Experiments
  10. Experiments of Panorama Generation
  11. Experiments of Panorama Reconstruction
  12. Limitations and Future Work
  13. Conclusion

Figures (10)

  • Figure 1: We propose HoloDreamer, a text-driven 3D scene generation framework to generate immersive and fully enclosed 3D scenes with high view-consistency. It consists of two basic modules: Stylized Equirectangular Panorama Generation, which generates a stylized high-quality equirectangular panorama from the input user prompt, and Enhanced Two-Stage Panorama Reconstruction, which employs 3D Gaussian Splatting for rapid 3D reconstruction of the panorama with enhanced integrity.
  • Figure 2: Overview of our Stylized Equirectangular Panorama Generation. Given a user prompt, multiple diffusion models are used to generate stylized high-quality panoramas. Additionally the circular blending technique is applied to avoid cracks when rotating the panorama.
  • Figure 3: Overview of our Enhanced Two-Stage Panorama Reconstruction. We perform depth estimation on the generated panorama and then project RGBD data to obtain the point cloud. Two types of cameras --- base cameras and supplementary cameras --- for projection and rendering in different scenarios, and prepare three image sets for supervision at different stages of 3D-GS optimization. The rendering images of the reconstructed scene from Pre Optimization stage are inpainted for optimization in Transfer Optimization stage, resulting in the final reconstructed scene.
  • Figure 4: Visual comparison of point cloud data: raw point cloud $\boldsymbol{P_0}$, depth filtered point cloud $\boldsymbol{P_f}$, and downsampled point cloud $\boldsymbol{P_s}$.
  • Figure 5: The relationship between the base camera and corresponding supplementary cameras as well as their projection results. Supplementary cameras surround the corresponding base camera, each with variations in position and orientation. The base camera is used for projection of the panorama, while the supplementary cameras are used for projecting point clouds.
  • ...and 5 more figures