SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs

TL;DR

SyntheOcc introduces a diffusion-based framework that conditions street-view image synthesis on 3D occupancy semantics using 3D semantic MPIs, enabling fine-grained geometric control and occlusion-aware conditioning for dataset generation. The method couples a 3D MPI representation with a lightweight MPI encoder and cross-view/cross-frame attention, augmented by targeted reweighting strategies, to produce photorealistic, geometry-consistent images across views and frames. Quantitative and qualitative results on nuScenes demonstrate improved realism and occupancy-alignment, and the synthetic data effectively augment downstream 3D occupancy prediction models while enabling corner-case scenario generation. This approach offers a scalable data engine for enhancing perception systems and simulating rare or hazardous conditions in autonomous driving contexts.

Abstract

The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric-controlled images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. As a result, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.

Figures (16)

• Figure 1: A showcase of application of SytheOcc. We enable geometric-controlled generation that conveys the user editing in 3D voxel space to generate realistic street view images. In this case, we create a rare scene that traffic cones block the way. This advancement facilitates the evaluation of autonomous systems, such as the end-to-end planner VAD jiang2023vad, in simulated corner case scenes.
• Figure 2: The overall architecture of SytheOcc. We achieve 3D geometric control in image generation by utilizing our proposed 3D semantic multiplane images to encode scene occupancy. In our framework, we can edit the occupied state and semantics of every voxel in 3D space to control the image generation, thereby opening up a wide spectrum of applications as shown in the top right.
• Figure 3: Visualizations of geometric controlled generation. Top row: Fusion of 3D semantic MPI. Bottom row: our generation concatenated from neighboring views.
• Figure 4: Visualizations of the reweighing function in Eq. \ref{['equ:reweight']}.
• Figure 5: Visualizations of generated multi-view images. The generation conditions (occupancy labels) are from nuScenes validation set. We highlight that (i) Geometry alignment of trees in red rectangle in (b). (ii) Use text prompt to control high-level appearance in (c,d).