Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method

TL;DR

This work tackles the scarcity of annotated occupancy data and the need for multi-modal driving scene synthesis by introducing Nuplan-Occ, the largest semantic occupancy dataset, and UniScenev2, a unified framework for 4D occupancy, multi-view video, and LiDAR generation. The approach decouples spatial expansion from temporal forecasting via a spatio-temporal disentangled architecture, and bridges modality gaps with a Gaussian splatting-based sparse point map rendering and a sensor-specific LiDAR embedding. Empirical results demonstrate superior generation fidelity across occupancy, video, and LiDAR tasks, with notable improvements in mIoU, FVD, and MMD metrics, and solid generalization to new sensor configurations. The combination of a scalable dataset and a hierarchical, occupancy-centric pipeline offers practical value for downstream autonomous driving tasks and broader multi-modal simulation.

Abstract

Driving scene generation is a critical domain for autonomous driving, enabling downstream applications, including perception and planning evaluation. Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities; however, their performance heavily depends on annotated occupancy data, which still remains scarce. To overcome this limitation, we curate Nuplan-Occ, the largest semantic occupancy dataset to date, constructed from the widely used Nuplan benchmark. Its scale and diversity facilitate not only large-scale generative modeling but also autonomous driving downstream applications. Based on this dataset, we develop a unified framework that jointly synthesizes high-quality semantic occupancy, multi-view videos, and LiDAR point clouds. Our approach incorporates a spatio-temporal disentangled architecture to support high-fidelity spatial expansion and temporal forecasting of 4D dynamic occupancy. To bridge modal gaps, we further propose two novel techniques: a Gaussian splatting-based sparse point map rendering strategy that enhances multi-view video generation, and a sensor-aware embedding strategy that explicitly models LiDAR sensor properties for realistic multi-LiDAR simulation. Extensive experiments demonstrate that our method achieves superior generation fidelity and scalability compared to existing approaches, and validates its practical value in downstream tasks. Repo: https://github.com/Arlo0o/UniScene-Unified-Occupancy-centric-Driving-Scene-Generation/tree/v2

Figures (16)

• Figure 1: Overview of Nuplan-Occ dataset and the UniScenev2 pipeline. We introduce the largest semantic occupancy dataset to date, featuring dense 3D semantic annotations that contain $\sim$19× more annotated scenes and $\sim$18× more frames than Nuscenes-Occupancy tian2024occ3dwei2023surroundocc. Facilitated with Nuplan-Occ, UniScenev2 scales up both model architecture and training data to enable high-quality occupancy spatial expansion and temporal forecasting, as well as occupancy-based sparse point map condition for video generation, and sensor-specific LiDAR generation.
• Figure 2: The Nuplan-Occ provides dense semantic occupancy labels for 10HZ all frames in the Nuplan Nuplan dataset. Compared with OpenScene openscene2023, our method demonstrates high resolution (400$\times$400$\times$32) dense annotations with accurate geometry (e.g., clear vehicle structures and smooth road surfaces).
• Figure 3: Nuplan-Occ dataset curation pipeline with the proposed Foreground-Background Separate Aggregation (FBSA) strategy. This strategy is composed of three key components: separated multi-frame point cloud aggregation, neural kernel-based mesh reconstruction, and hybrid semantic labeling.
• Figure 4: Overall framework of UniScenev2. The joint generation process facilitates large-scale dynamic generation with an occupancy-centric hierarchy: I. Dynamic Large-scale Occupancy Generation. The optional BEV layout is concatenated with the noise volume before being fed into the occupancy spatial diffusion transformer, and decoded with the occupancy VAE decoder $\mathcal{D}_\mathrm{occ}$ to generate large-scale occupancy grids. The occupancy temporal diffusion transformer processes a selected occupancy scene to forecast temporal occupancy sequences. II. Occupancy-based Multi-view Video and LiDAR Generation. The occupancy is converted into 3D Gaussians and rendered into sparse semantic and depth point maps, which guide the video generation with a video diffusion transformer. The output is obtained from the video VAE decoder $\mathcal{D}_\mathrm{vid}$. For LiDAR generation, the sparse LiDAR UNet takes occupancy grids and sensor rig data as inputs, which are then passed to the LiDAR head $\mathcal{D}_\mathrm{lid}$ for multi-view LiDAR generation.
• Figure 5: (a) Architecture of the occupancy generation model, which integrates a 4D occupancy VAE and an occupancy Diffusion Transformer (DiT). (b) Spatio-temporal Disentangled Generation pipeline.