PanoDreamer: Consistent Text to 360-Degree Scene Generation

TL;DR

PanoDreamer introduces a two-stage framework for consistent text- and image-guided 360-degree scene generation: first synthesize a complete panorama guided by an LLM and warping, then lift to 3D via depth-estimated point clouds and 3D Gaussian Splatting. The method extends static panoramas with trajectory-guided moving views using conditional diffusion, followed by depth-aligned fusion and a two-stage refinement of 3D Gaussians to ensure geometric consistency across viewpoints. Key contributions include semantically preserved warping for occlusion filling, depth alignment between panorama and moving scenes, and a global-level 3D reconstruction that supports free camera movement. Experiments show higher-quality, more coherent 360-degree scenes than baselines, with robust performance across indoor and outdoor environments and customizable scene extensions, enabling practical VR/AR and gaming applications.

Abstract

Automatically generating a complete 3D scene from a text description, a reference image, or both has significant applications in fields like virtual reality and gaming. However, current methods often generate low-quality textures and inconsistent 3D structures. This is especially true when extrapolating significantly beyond the field of view of the reference image. To address these challenges, we propose PanoDreamer, a novel framework for consistent, 3D scene generation with flexible text and image control. Our approach employs a large language model and a warp-refine pipeline, first generating an initial set of images and then compositing them into a 360-degree panorama. This panorama is then lifted into 3D to form an initial point cloud. We then use several approaches to generate additional images, from different viewpoints, that are consistent with the initial point cloud and expand/refine the initial point cloud. Given the resulting set of images, we utilize 3D Gaussian Splatting to create the final 3D scene, which can then be rendered from different viewpoints. Experiments demonstrate the effectiveness of PanoDreamer in generating high-quality, geometrically consistent 3D scenes.

Figures (7)

• Figure 1: Modules of our proposed framework. (a) Text-to-Panorama Generation: we use LLM as guidance to guide the generation of perspective-view images. (b) Scene Generation: we divide it into static panorama scene generation and customized moving scene generation. Besides base camera set, we compose an additional supplementary camera set for the static panorama scene and use semantic-preserved warping to generate the missing region, which is used for 3d gaussian splatting refinement.
• Figure 2: Semantic-preserved Refinement: For each base camera, we apply supplementary cameras to up, down, left, and right directions respectively. For each supplementary camera, we get projected images through semantic-preserved generative warping seo2024genwarp to fill the missing area brought by occlusion.
• Figure 3: Comparison of results. For Text2Room the images are projected from mesh and for LucidDreamer and our method, the images are projected from 3D Gaussians. Text2Room fail in generating specific stylized scenes. Compared with Text2Room and LucidDreamer, our method shows less artifacts and better geometry consistency.
• Figure 4: Comparison of Text-to-Panorama Generation. As panoramas generated by MultiDiffusion bar2023multidiffusion and MVDiffusion Tang2023mvdiffusion have both limited vertical FoV, for comparison, we only show our panorama before outpainting. Compared with previous methods, our method shows less duplicated objects and better generation quality.
• Figure 5: Our rendered rendered images with corresponding rendered depth.