FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

TL;DR

FUTR3D addresses the challenge of flexible 3D object detection across arbitrary sensor configurations by introducing a unified end-to-end fusion framework. It cores on a Modality-Agnostic Feature Sampler (MAFS) that samples and aggregates features from cameras, high- and low-resolution LiDARs, and radars at 3D reference points, followed by a shared transformer decoder with iterative refinement and a set-to-set loss for one-to-one predictions. The approach demonstrates strong performance on nuScenes across modalities, including a notable result of 58.0 $mAP$ with a 4-beam LiDAR and cameras, outperforming CenterPoint with a 32-beam LiDAR, and shows robust behavior with low-cost sensor setups. By removing reliance on hand-tuned late fusion heuristics, FUTR3D offers a scalable, cost-effective solution for autonomous perception and sets a foundation for future multi-modal fusion research.

Abstract

Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs depending on the sensor combinations or setups. In this work, we propose the first unified end-to-end sensor fusion framework for 3D detection, named FUTR3D, which can be used in (almost) any sensor configuration. FUTR3D employs a query-based Modality-Agnostic Feature Sampler (MAFS), together with a transformer decoder with a set-to-set loss for 3D detection, thus avoiding using late fusion heuristics and post-processing tricks. We validate the effectiveness of our framework on various combinations of cameras, low-resolution LiDARs, high-resolution LiDARs, and Radars. On NuScenes dataset, FUTR3D achieves better performance over specifically designed methods across different sensor combinations. Moreover, FUTR3D achieves great flexibility with different sensor configurations and enables low-cost autonomous driving. For example, only using a 4-beam LiDAR with cameras, FUTR3D (58.0 mAP) achieves on par performance with state-of-the-art 3D detection model CenterPoint (56.6 mAP) using a 32-beam LiDAR.

Figures (5)

• Figure 1: Results visualization in BEV. On the left is 1-beam LiDAR with cameras, in the middle is 4-beam LiDAR with cameras and on the right is 32-beam LiDAR with cameras.
• Figure 2: Overview of FUTR3D. Each sensor modality is encoded individually in its own coordinate. Then a query-based Modality-Agnostic Feature Sampler (MAFS) extracts features from all available modalities according to the 3D reference point of each query. Finally a transformer decoder predicts 3D bounding boxes from queries. The predicted boxes can be iteratively fed back into MAFS and transformer decoder to refine the predictions.
• Figure 2: Results visualization in perspective images. On the left is 4-beam LiDAR with cameras; on the right is 32-beam LiDAR. FUTR3D with 4-beam LiDAR with cameras achieves competitive performance compared to 32-beam LiDAR, especially for small objects like pedestrians and bicycles, and objects in the distance.
• Figure 3: Qualitative results of FUTR3D. We show perspective image view results by projecting LiDAR points onto images. (a) There is a car in the distance marked in red circle which are missed by 32-beam LiDAR based detector. (b) The billboard circled in red is detected falsely as pedestrian using vision only. This can be corrected with the help of 1-beam LiDAR.
• Figure 3: Results visualization in perspective images. On the left is 1-beam LiDAR with cameras, on the right is cameras. Even sparse points like 1-beam LiDAR can help FUTR3D to detect and correct the false positive.