BEVDiffuser: Plug-and-Play Diffusion Model for BEV Denoising with Ground-Truth Guidance

TL;DR

BEVDiffuser introduces a diffusion-based BEV denoising model guided by ground-truth object layouts, designed as a training-time plug-in for existing BEV encoders with no inference-time overhead. By conditioning denoising on a layout layout-derived embedding via a LayoutDiffusion fusion module, it jointly optimizes diffusion denoising and downstream-task losses to produce cleaner BEV feature maps. On nuScenes, BEVDiffuser yields substantial improvements in 3D object detection metrics (e.g., mAP and NDS) across multiple BEV backbones and shows stronger performance on long-tail objects and under adverse weather, while also supporting controllable BEV generation from layouts. The approach demonstrates the practicality of diffusion-based BEV denoising as a data-augmentationlike training aid, enabling more robust BEV representations without changing deployment architectures or adding inference latency.

Abstract

Bird's-eye-view (BEV) representations play a crucial role in autonomous driving tasks. Despite recent advancements in BEV generation, inherent noise, stemming from sensor limitations and the learning process, remains largely unaddressed, resulting in suboptimal BEV representations that adversely impact the performance of downstream tasks. To address this, we propose BEVDiffuser, a novel diffusion model that effectively denoises BEV feature maps using the ground-truth object layout as guidance. BEVDiffuser can be operated in a plug-and-play manner during training time to enhance existing BEV models without requiring any architectural modifications. Extensive experiments on the challenging nuScenes dataset demonstrate BEVDiffuser's exceptional denoising and generation capabilities, which enable significant enhancement to existing BEV models, as evidenced by notable improvements of 12.3\% in mAP and 10.1\% in NDS achieved for 3D object detection without introducing additional computational complexity. Moreover, substantial improvements in long-tail object detection and under challenging weather and lighting conditions further validate BEVDiffuser's effectiveness in denoising and enhancing BEV representations.

Figures (11)

• Figure 1: Comparisons of BEV feature maps: (a) generated by BEVFormer (tiny) bevformer, (b) denoised by BEVDiffuser in $5$ steps. Channel-wise features are averaged for visualization. BEVDiffuser denoises and substantially enhances the BEV feature maps.
• Figure 2: Left: A sketch of common BEV models that generate BEV feature maps from sensor inputs through a BEV encoder. BEV feature maps are usually optimized for downstream task performance. Right: Overview of BEVDiffuser, which consists of a U-Net that predicts the clean BEV features from the noisy ones, conditioned on the ground-truth layout. It is trained on BEV feature maps produced by BEV models with multiple steps of noise added, and is optimized using a joint loss composed of a diffusion loss and a downstream task loss.
• Figure 3: BEVDiffuser can be plugged into the training process of a BEV model. It denoises the BEV feature maps produced by existing BEV encoders over $K$ steps and provides the denoised BEV as supervision for BEV predictions.
• Figure 4: 3D object detection performance of various BEV models on nuScenes val dataset (denoising steps $=0$). The performance ramps up when adopting BEVDiffuser to denoise their BEV feature maps with increasing denoising steps, indicating the powerful denoising capability of our BEVDiffuser.
• Figure 5: 3D object detection visualizations of two BEV feature maps generated by our BEVDiffuser ($\mathrm{BD}^{fu}$) from random noise. The alignment between predictions and ground truth demonstrates that BEVDiffuser has strong controllable generation capability.