DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion

TL;DR

DualDiff addresses the challenge of conditioning autonomous driving scene generation on rich multi-modal information. It introduces a dual-branch diffusion framework that leverages Occupancy Ray Sampling (ORS) and numerical driving-scene representations, fused through Semantic Fusion Attention to align modalities. A foreground-aware masked loss emphasizes tiny object fidelity. The method achieves state-of-the-art Fréchet Inception Distance and improves BEV segmentation and 3D object detection, with successful transfer to Waymo and enhanced downstream perception training using synthetic data.

Abstract

Accurate and high-fidelity driving scene reconstruction relies on fully leveraging scene information as conditioning. However, existing approaches, which primarily use 3D bounding boxes and binary maps for foreground and background control, fall short in capturing the complexity of the scene and integrating multi-modal information. In this paper, we propose DualDiff, a dual-branch conditional diffusion model designed to enhance multi-view driving scene generation. We introduce Occupancy Ray Sampling (ORS), a semantic-rich 3D representation, alongside numerical driving scene representation, for comprehensive foreground and background control. To improve cross-modal information integration, we propose a Semantic Fusion Attention (SFA) mechanism that aligns and fuses features across modalities. Furthermore, we design a foreground-aware masked (FGM) loss to enhance the generation of tiny objects. DualDiff achieves state-of-the-art performance in FID score, as well as consistently better results in downstream BEV segmentation and 3D object detection tasks.

Figures (6)

• Figure 1: We have achieved state-of-the-art performance in several evaluation metrics compared to other custom or base models. To present the data in the charts more clearly, we have scaled some of the metrics.
• Figure 2: Overview of DualDiff for multi-view image generation. We use occupancy ray sampling (ORS) and numerical driving scene representation, which are fused through the proposed semantic fusion attention (SFA) module and then used as inputs to the dual-branch foreground-background architecture. The outputs of the branches are then merged back into the UNet in the form of ControlNet residuals to obtain the final output.
• Figure 3: Illustrations of our proposed Semantic Fusion Attention (SFA), which sequentially fuses ORS features with multi-modal information.
• Figure 4: Driving scenes of (a) ground truth, (b) MagicDrive and (c) DualDiff (Ours). Compared to the baseline, DualDiff faithfully reproduces the left turn orientation as well as the car in distance in the night scene, while in the daylight case, DualDiff generates the edge of the road as well as the tree behind precisely.
• Figure 5: Reconstruction scene in daylight, where our model generates the bus in distance and the lamp pole correctly.