Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Deep Learning Based Radar and Vision Fusion for 3D Object Detection in Autonomous Driving

Di Wu, Feng Yang, Benlian Xu, Pan Liao, Bo Liu

TL;DR

The paper addresses robust 3D object detection in autonomous driving by surveying deep learning-based radar-vision fusion approaches. It categorizes methods into ROI-based and end-to-end fusion, with end-to-end further split into 3D bbox prediction-based and BEV-based strategies, and it surveys the latest 4D radar developments for AVs. The review covers datasets, representative fusion architectures (including attention-based cross-modal fusion), and depth-aware BEV techniques, while highlighting challenges such as radar sparsity and cross-modal alignment. It concludes with anticipated trends including end-to-end driving, expanded use of 4D radar, and collaborative perception to enhance safety and scalability in real-world deployments.

Abstract

With the rapid advancement of autonomous driving technology, there is a growing need for enhanced safety and efficiency in the automatic environmental perception of vehicles during their operation. In modern vehicle setups, cameras and mmWave radar (radar), being the most extensively employed sensors, demonstrate complementary characteristics, inherently rendering them conducive to fusion and facilitating the achievement of both robust performance and cost-effectiveness. This paper focuses on a comprehensive survey of radar-vision (RV) fusion based on deep learning methods for 3D object detection in autonomous driving. We offer a comprehensive overview of each RV fusion category, specifically those employing region of interest (ROI) fusion and end-to-end fusion strategies. As the most promising fusion strategy at present, we provide a deeper classification of end-to-end fusion methods, including those 3D bounding box prediction based and BEV based approaches. Moreover, aligning with recent advancements, we delineate the latest information on 4D radar and its cutting-edge applications in autonomous vehicles (AVs). Finally, we present the possible future trends of RV fusion and summarize this paper.

A Survey of Deep Learning Based Radar and Vision Fusion for 3D Object Detection in Autonomous Driving

TL;DR

The paper addresses robust 3D object detection in autonomous driving by surveying deep learning-based radar-vision fusion approaches. It categorizes methods into ROI-based and end-to-end fusion, with end-to-end further split into 3D bbox prediction-based and BEV-based strategies, and it surveys the latest 4D radar developments for AVs. The review covers datasets, representative fusion architectures (including attention-based cross-modal fusion), and depth-aware BEV techniques, while highlighting challenges such as radar sparsity and cross-modal alignment. It concludes with anticipated trends including end-to-end driving, expanded use of 4D radar, and collaborative perception to enhance safety and scalability in real-world deployments.

Abstract

With the rapid advancement of autonomous driving technology, there is a growing need for enhanced safety and efficiency in the automatic environmental perception of vehicles during their operation. In modern vehicle setups, cameras and mmWave radar (radar), being the most extensively employed sensors, demonstrate complementary characteristics, inherently rendering them conducive to fusion and facilitating the achievement of both robust performance and cost-effectiveness. This paper focuses on a comprehensive survey of radar-vision (RV) fusion based on deep learning methods for 3D object detection in autonomous driving. We offer a comprehensive overview of each RV fusion category, specifically those employing region of interest (ROI) fusion and end-to-end fusion strategies. As the most promising fusion strategy at present, we provide a deeper classification of end-to-end fusion methods, including those 3D bounding box prediction based and BEV based approaches. Moreover, aligning with recent advancements, we delineate the latest information on 4D radar and its cutting-edge applications in autonomous vehicles (AVs). Finally, we present the possible future trends of RV fusion and summarize this paper.
Paper Structure (16 sections, 3 equations, 4 figures, 2 tables)

This paper contains 16 sections, 3 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Tasks
  4. Datasets
  5. Others
  6. 3D Object Detection Based on Radar-Vision Fusion
  7. ROI-Based Fusion
  8. End-to-End Fusion
  9. 3D Bbox Prediction based
  10. BEV based
  11. 4D Radar Applications in AVs
  12. Future Trends
  13. End-to-End Autonomous Driving
  14. Application of 4D Radar
  15. Collaborative Perception
  16. ...and 1 more sections

Figures (4)

  • Figure 1: The organization of this paper.
  • Figure 2: 2D and 3D object detection results. Left: 2D object detectionli2020feature. Right: 3D object detectionli2022bevformer.
  • Figure 3: The ROI-based RV fusion frameworks ((a): Radar-generated ROIs; (b): Vision-generated ROIs).
  • Figure 4: The general end-to-end RV fusion framework.