Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection

Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, Feng Zhao

TL;DR

BEVDistill introduces a cross-modal distillation framework that transfers knowledge from a LiDAR-based teacher to a camera-based student in BEV space for multi-view 3D object detection. It couples dense foreground-guided feature distillation with sparse mutual-information-based instance distillation to bridge modality gaps without adding inference cost. Empirical results on nuScenes demonstrate significant improvements over strong baselines and establish new state-of-the-art performance, including 59.4 NDS on the nuScenes test. The approach is robust across backbone choices and datasets, highlighting the practical potential of BEV-space cross-modal distillation for efficient 3D detection in real-world deployments.

Abstract

3D object detection from multiple image views is a fundamental and challenging task for visual scene understanding. Owing to its low cost and high efficiency, multi-view 3D object detection has demonstrated promising application prospects. However, accurately detecting objects through perspective views is extremely difficult due to the lack of depth information. Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deployment. Different from the images, LiDAR points are superior in providing spatial cues, resulting in highly precise localization. In this paper, we explore the incorporation of LiDAR-based detectors for multi-view 3D object detection. Instead of directly training a depth prediction network, we unify the image and LiDAR features in the Bird-Eye-View (BEV) space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm. To this end, we propose \textbf{BEVDistill}, a cross-modal BEV knowledge distillation (KD) framework for multi-view 3D object detection. Extensive experiments demonstrate that the proposed method outperforms current KD approaches on a highly-competitive baseline, BEVFormer, without introducing any extra cost in the inference phase. Notably, our best model achieves 59.4 NDS on the nuScenes test leaderboard, achieving new state-of-the-art in comparison with various image-based detectors. Code will be available at https://github.com/zehuichen123/BEVDistill.

BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection

TL;DR

BEVDistill introduces a cross-modal distillation framework that transfers knowledge from a LiDAR-based teacher to a camera-based student in BEV space for multi-view 3D object detection. It couples dense foreground-guided feature distillation with sparse mutual-information-based instance distillation to bridge modality gaps without adding inference cost. Empirical results on nuScenes demonstrate significant improvements over strong baselines and establish new state-of-the-art performance, including 59.4 NDS on the nuScenes test. The approach is robust across backbone choices and datasets, highlighting the practical potential of BEV-space cross-modal distillation for efficient 3D detection in real-world deployments.

Abstract

3D object detection from multiple image views is a fundamental and challenging task for visual scene understanding. Owing to its low cost and high efficiency, multi-view 3D object detection has demonstrated promising application prospects. However, accurately detecting objects through perspective views is extremely difficult due to the lack of depth information. Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deployment. Different from the images, LiDAR points are superior in providing spatial cues, resulting in highly precise localization. In this paper, we explore the incorporation of LiDAR-based detectors for multi-view 3D object detection. Instead of directly training a depth prediction network, we unify the image and LiDAR features in the Bird-Eye-View (BEV) space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm. To this end, we propose \textbf{BEVDistill}, a cross-modal BEV knowledge distillation (KD) framework for multi-view 3D object detection. Extensive experiments demonstrate that the proposed method outperforms current KD approaches on a highly-competitive baseline, BEVFormer, without introducing any extra cost in the inference phase. Notably, our best model achieves 59.4 NDS on the nuScenes test leaderboard, achieving new state-of-the-art in comparison with various image-based detectors. Code will be available at https://github.com/zehuichen123/BEVDistill.
Paper Structure (29 sections, 15 equations, 2 figures, 12 tables)

This paper contains 29 sections, 15 equations, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-based 3D Object Detection
  4. Knowledge Distillation in Object Detection
  5. Methods
  6. Baseline Model
  7. BEVDistill
  8. Dense Feature Distillation
  9. Sparse Instance Distillation
  10. Experiments
  11. Dataset and Evaluation Metrics
  12. Implementation Details
  13. Main Results
  14. Comparison with State-of-the-Arts
  15. Ablation Studies
  16. ...and 14 more sections

Figures (2)

  • Figure 1: The overall architecture of the proposed BEVDistill consists of three main components: a transformer-based LiDAR detector, a transformer-based image detector, and the proposed cross-modal distillation modules. It follows the teacher-student paradigm with both RGB images and LiDAR points as the inputs and keeps the original structure of respective model structures.
  • Figure 2: Visualization of the detection results predicted by BEVDistill. Best view in color: the prediction and ground truth are in blue and orange, respectively.