CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation

TL;DR

Outdoor 3D semantic scene generation has been data-constrained and hard to control. This work introduces SketchSem3D, a public multi-condition benchmark, and CymbaDiff, a diffusion-based generator operating in a VAE latent space with Cylinder Mamba blocks to enforce angular continuity and vertical hierarchy. Experiments on SketchSem3D show state-of-the-art semantic consistency, spatial realism, and strong cross-dataset generalization to KITTI-360, establishing a solid baseline for future outdoor 3D generation research. The dataset and method together enable scalable urban-scale simulation and autonomous-driving research with intuitive sketch- and PSA-driven conditioning.

Abstract

Outdoor 3D semantic scene generation produces realistic and semantically rich environments for applications such as urban simulation and autonomous driving. However, advances in this direction are constrained by the absence of publicly available, well-annotated datasets. We introduce SketchSem3D, the first large-scale benchmark for generating 3D outdoor semantic scenes from abstract freehand sketches and pseudo-labeled annotations of satellite images. SketchSem3D includes two subsets, Sketch-based SemanticKITTI and Sketch-based KITTI-360 (containing LiDAR voxels along with their corresponding sketches and annotated satellite images), to enable standardized, rigorous, and diverse evaluations. We also propose Cylinder Mamba Diffusion (CymbaDiff) that significantly enhances spatial coherence in outdoor 3D scene generation. CymbaDiff imposes structured spatial ordering, explicitly captures cylindrical continuity and vertical hierarchy, and preserves both physical neighborhood relationships and global context within the generated scenes. Extensive experiments on SketchSem3D demonstrate that CymbaDiff achieves superior semantic consistency, spatial realism, and cross-dataset generalization. The code and dataset will be available at https://github.com/Lillian-research-hub/CymbaDiff

Figures (5)

• Figure 1: Pipeline for SketchSem3D construction. SAM and PSA denote Segment Anything Model and Pseudo-labeled Satellite Image Annotations, respectively.
• Figure 2: Architecture of our CymbaDiff generation network. The Scene Structure Estimation Network (SSEN) extracts abstract structural information from Pseudo-labeled Satellite Image Annotations (PSA) and the Sketch. The Latent Mapping Network (LMN) compresses the input conditions into a latent representation, which is then processed by the CymbaDiff denoiser, which utilizes the proposed cylinder mamba blocks (CylMa) to perform latent denoising.
• Figure 3: Architecture of the CymbaDiff denoising network. CylMamba denotes cylinder mamba block. Refer to the text for details.
• Figure 4: Qualitative results on the Sketch-based SemanticKITTI validation set. The 1st and 2nd rows show generated scenes conditioned on the corresponding freehand sketch and pseudo-labeled satellite images. The 3rd and 4th rows demonstrate the model’s capability to generate moderately diverse 3D scenes (with different details) under identical input conditions.
• Figure 5: Qualitative results on the SemanticKITTI validation set. Columns from the left represent ground truth, and outputs of CymbaDiff (our method), MonoScene, OccFormer, and VoxFormer.