Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

Keli Huang, Botian Shi, Xiang Li, Xin Li, Siyuan Huang, Yikang Li

TL;DR

The paper addresses robust perception for autonomous driving by integrating multi-modal data from LiDAR and cameras. It introduces a novel taxonomy dividing fusion methods into strong-fusion (with early, deep, late, and asymmetry) and weak-fusion, then surveys input representations and major datasets. It analyzes remaining challenges—misalignment, information loss, domain bias, and resolution mismatches—and outlines directions such as advanced fusion operations, temporal/self-supervised learning, and multi-source information leverage. Overall, the work provides a systematic framework to guide future fusion research and practical deployment in diverse driving scenarios.

Abstract

Multi-modal fusion is a fundamental task for the perception of an autonomous driving system, which has recently intrigued many researchers. However, achieving a rather good performance is not an easy task due to the noisy raw data, underutilized information, and the misalignment of multi-modal sensors. In this paper, we provide a literature review of the existing multi-modal-based methods for perception tasks in autonomous driving. Generally, we make a detailed analysis including over 50 papers leveraging perception sensors including LiDAR and camera trying to solve object detection and semantic segmentation tasks. Different from traditional fusion methodology for categorizing fusion models, we propose an innovative way that divides them into two major classes, four minor classes by a more reasonable taxonomy in the view of the fusion stage. Moreover, we dive deep into the current fusion methods, focusing on the remaining problems and open-up discussions on the potential research opportunities. In conclusion, what we expect to do in this paper is to present a new taxonomy of multi-modal fusion methods for the autonomous driving perception tasks and provoke thoughts of the fusion-based techniques in the future.

Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

TL;DR

The paper addresses robust perception for autonomous driving by integrating multi-modal data from LiDAR and cameras. It introduces a novel taxonomy dividing fusion methods into strong-fusion (with early, deep, late, and asymmetry) and weak-fusion, then surveys input representations and major datasets. It analyzes remaining challenges—misalignment, information loss, domain bias, and resolution mismatches—and outlines directions such as advanced fusion operations, temporal/self-supervised learning, and multi-source information leverage. Overall, the work provides a systematic framework to guide future fusion research and practical deployment in diverse driving scenarios.

Abstract

Multi-modal fusion is a fundamental task for the perception of an autonomous driving system, which has recently intrigued many researchers. However, achieving a rather good performance is not an easy task due to the noisy raw data, underutilized information, and the misalignment of multi-modal sensors. In this paper, we provide a literature review of the existing multi-modal-based methods for perception tasks in autonomous driving. Generally, we make a detailed analysis including over 50 papers leveraging perception sensors including LiDAR and camera trying to solve object detection and semantic segmentation tasks. Different from traditional fusion methodology for categorizing fusion models, we propose an innovative way that divides them into two major classes, four minor classes by a more reasonable taxonomy in the view of the fusion stage. Moreover, we dive deep into the current fusion methods, focusing on the remaining problems and open-up discussions on the potential research opportunities. In conclusion, what we expect to do in this paper is to present a new taxonomy of multi-modal fusion methods for the autonomous driving perception tasks and provoke thoughts of the fusion-based techniques in the future.
Paper Structure (18 sections, 8 figures, 3 tables)

This paper contains 18 sections, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Tasks, and Open Competitions
  3. Multi-modal Sensor Fusion Perception Tasks
  4. Open competitions and Datasets
  5. Representations for LiDAR and Image
  6. Image Representation
  7. Point-based Point Cloud Representation
  8. Voxel-based Point Cloud Representation
  9. 2D-mapping-based Point Cloud Representation
  10. Fusion Methodology
  11. Strong-fusion
  12. Weak-fusion
  13. Other Fusion Methods
  14. Opportunities in Multi-Modal Fusion
  15. More Advanced Fusion Methodology
  16. ...and 3 more sections

Figures (8)

  • Figure 1: Perception Tasks of Autonomous Driving by Multi-modal Sensor Fusion Model.
  • Figure 2: Fusion Methodology Overview
  • Figure 3: Strong-Fusion Overview
  • Figure 4: An Example of Early-Fusion
  • Figure 5: An Example of Deep-Fusion
  • ...and 3 more figures