HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation

TL;DR

HiScene addresses the challenge of scene-level 3D generation by proposing a hierarchical framework that treats rooms as complex objects and inner items as manipulatable sub-objects under isometric views. It integrates a native 3D generator (TRELLIS) with hierarchical scene parsing and a video-diffusion-based amodal completion to recover complete object identities under occlusion, aided by shadow-aware data and a shape-prior refinement step for spatial alignment. The method yields compositional, editable 3D scenes that align with 2D prompts and support interactive editing, demonstrated via improved text-scene alignment, object completeness, and layout realism against baselines. This work advances practical scene synthesis with better object coherence, spatial plausibility, and cross-domain consistency, enabling applications in interactive editing, simulation, and robotics.

Abstract

Scene-level 3D generation represents a critical frontier in multimedia and computer graphics, yet existing approaches either suffer from limited object categories or lack editing flexibility for interactive applications. In this paper, we present HiScene, a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation and delivers high-fidelity scenes with compositional identities and aesthetic scene content. Our key insight is treating scenes as hierarchical "objects" under isometric views, where a room functions as a complex object that can be further decomposed into manipulatable items. This hierarchical approach enables us to generate 3D content that aligns with 2D representations while maintaining compositional structure. To ensure completeness and spatial alignment of each decomposed instance, we develop a video-diffusion-based amodal completion technique that effectively handles occlusions and shadows between objects, and introduce shape prior injection to ensure spatial coherence within the scene. Experimental results demonstrate that our method produces more natural object arrangements and complete object instances suitable for interactive applications, while maintaining physical plausibility and alignment with user inputs.

Figures (16)

• Figure 1: HiScene allows users to generate scene-level 3D assets with natural layout and appealing looking, while delivering compositional items for versatile applications such as interactive editing and simulation.
• Figure 2: Overview of HiScene. Our hierarchical framework generates 3D scenes with compositional identities through three main stages. First, we create a 3D scene from a generated isometric view. Next, we perform scene parsing to obtain precise object segmentation, followed by multi-view rendering and detailed occlusion analysis for each identified instance. Finally, we apply our video-diffusion-based amodal completion to generate complete views of each instance, which serve as guidance for regenerating intact objects with proper spatial alignment in the scene. The resulting 3D scene features fully compositional identities, facilitating user-directed modifications like interactive scene editing.
• Figure 3: Comparison of perspective view and isometric view of a living room scene. Zoom in for more details.
• Figure 4: We present an data curation example of amodal completion, including original image (a), occluded input image (b), visible mask (c), and the linear blended video (d). We also present shadow-aware data examples (e).
• Figure 5: An illustration of Spatial Aligned Generation. We use sparse-view LRM to initialize spatial aligned shape prior (voxel latent), and inject this prior by initializing voxel noises upon it during native 3D generation, thus ensuring regenerated assets adhering the original scene.