Enhanced Automotive Object Detection via RGB-D Fusion in a DiffusionDet Framework

TL;DR

This work extends DiffusionDet to RGB-D data for automotive object detection by introducing FusedDiffusionDet, a two-stream architecture that fuses RGB and depth features through mid-level concatenation and diffusion-based bounding-box refinement. The model employs dual ResNet-50 backbones with feature pyramids, a diffusion forward–reverse process, and a loss combining focal classification with L1 and generalized IoU regression, trained on KITTI with pre-trained backbones. Key findings show a significant improvement for small objects (3.7% AP) and large objects (2.9% AP) when using RGB-D fusion, with the best results achieved by concatenation fusion at 512 output channels, outperforming the RGB baseline while maintaining efficiency. The results demonstrate that RGB-D fusion can enhance robustness and accuracy in urban driving scenarios, suggesting practical benefits for ADAS and autonomous driving systems, and indicating broader applicability to other 2D sensor modalities.

Abstract

Vision-based autonomous driving requires reliable and efficient object detection. This work proposes a DiffusionDet-based framework that exploits data fusion from the monocular camera and depth sensor to provide the RGB and depth (RGB-D) data. Within this framework, ground truth bounding boxes are randomly reshaped as part of the training phase, allowing the model to learn the reverse diffusion process of noise addition. The system methodically enhances a randomly generated set of boxes at the inference stage, guiding them toward accurate final detections. By integrating the textural and color features from RGB images with the spatial depth information from the LiDAR sensors, the proposed framework employs a feature fusion that substantially enhances object detection of automotive targets. The $2.3$ AP gain in detecting automotive targets is achieved through comprehensive experiments using the KITTI dataset. Specifically, the improved performance of the proposed approach in detecting small objects is demonstrated.

Figures (6)

• Figure 1: An RGB-D diffusion model for object detection. (a) A diffusion model where $q$ is the diffusion process and $p_\theta$ is the learnable reverse process. (b) Diffusion model for image generation task. (c) Object detection is a denoising diffusion process from noisy boxes to object boxes on the RGB-D dataset.
• Figure 2: Schematic representation of the proposed diffusion-based object detection architecture using RGB-D data. A $3$D point cloud is transformed into a $2$D image, enriching it with depth information. The diffusion detection model chen2023diffusiondet network uses fused features to detect and classify objects. The output is a set of bounding boxes that indicate the location and classification of objects within the scene.
• Figure 3: Feature fusion processing, where the green nodes denote fusion operation.
• Figure 4: Comparison of training loss functions with random and pre-trained backbones. A pre-trained backbone significantly enhances performance and reduces computational time compared to the initialization of random weights. Therefore, all our experiments use pre-trained backbones.
• Figure 5: Training loss functions for RGB and low-level fused RGB-D models. Low-level fusion does not improve detection performance. Notice that ${\cal N}_{c \oplus d}^{256}$ with a partial usage of a pre-trained backbone model outperforms ${\cal N}_{c \oplus d}^{s,256}$.