DeepInteraction++: Multi-Modality Interaction for Autonomous Driving

TL;DR

This work addresses the limitation of fusion-based multi-modal perception in autonomous driving by preserving two modality-specific representations (LiDAR BEV and camera image) throughout the perception pipeline and enabling information exchange via modality interaction. It introduces a dual-stream Transformer encoder with multi-modal representational interaction (MMRI) and a multi-modal predictive interaction (MMPI) decoder, augmented by LiDAR-guided cross-plane polar attention and grouped sparse attention for efficiency. The approach achieves state-of-the-art results on the nuScenes benchmark for 3D object detection and extends gracefully to end-to-end autonomous driving tasks, including segmentation, motion prediction, and planning, with demonstrated improvements in perception and planning metrics ($mAP$ and $NDS$). Extensive ablations validate the contribution of each component, confirming that the bilateral interaction between modality-specific representations yields robust, scalable performance across diverse LiDAR backbones and tasks, signaling a practical path toward more reliable multi-modal autonomous systems.

Abstract

Existing top-performance autonomous driving systems typically rely on the multi-modal fusion strategy for reliable scene understanding. This design is however fundamentally restricted due to overlooking the modality-specific strengths and finally hampering the model performance. To address this limitation, in this work, we introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout, enabling their unique characteristics to be exploited during the whole perception pipeline. To demonstrate the effectiveness of the proposed strategy, we design DeepInteraction++, a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Specifically, the encoder is implemented as a dual-stream Transformer with specialized attention operation for information exchange and integration between separate modality-specific representations. Our multi-modal representational learning incorporates both object-centric, precise sampling-based feature alignment and global dense information spreading, essential for the more challenging planning task. The decoder is designed to iteratively refine the predictions by alternately aggregating information from separate representations in a unified modality-agnostic manner, realizing multi-modal predictive interaction. Extensive experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks. Our code is available at https://github.com/fudan-zvg/DeepInteraction.

Figures (9)

• Figure 1: Schematic strategy comparison. (a) Existing multi-modality fusion-based 3D detection: Fusing individual per-modality representations into a single hybrid representation from which the detection results are further decoded. (b) Our multi-modality interaction-based 3D detection: Maintaining two modality-specific representations throughout the whole pipeline with both representational interaction in the encoder and predictive interaction in the decoder.
• Figure 2: Structure of our multi-modal representational interaction encoder. (a) Overall architecture: Given two modality-specific representations, the image-to-LiDAR feature interaction (b) spreads the visual signal in the image representation to the LiDAR BEV representation, and the LiDAR-to-image feature interaction (c) takes cross-modal relative contexts from LiDAR representation to enhance the image representations.
• Figure 3: Illustration of our multi-modal predictive interaction. Our predictive interaction decoder (a) generates predictions via (b) progressively interacting with two modality-specific representations.
• Figure 4: Qualitative results on nuScenes val set. In LiDAR BEV (right), green boxes are the ground-truth and blue boxes are the predictions. Best viewed when zooming in.
• Figure 5: Illustrations of the heatmaps predicted from BEV representations before (top) and after (bottom) representational interactions. All samples are from the nuScenes val split. (a) Occluded tiny objects. (b) Small objects at long distance. (c) Adjacent barriers connecting together in LiDAR point clouds thus difficult to discriminate without the help of visual clues.