IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection

TL;DR

IS-Fusion tackles the challenge of sparse, small-object BEV representations in multimodal 3D object detection by introducing instance-level collaboration alongside scene-level fusion. The method comprises Hierarchical Scene Fusion (HSF) to capture multi-granularity scene context via Point-to-Grid and Grid-to-Region transformers and Instance-Guided Fusion (IGF) to mine and reason about foreground instances, aggregating multimodal context with deformable attention and an Instance-to-Scene transformer. Together, these modules produce an instance-aware BEV representation that improves 3D detection accuracy, achieving state-of-the-art results on nuScenes with notable gains over BEVFusion and other multimodal methods. The work demonstrates the practical impact of explicit instance–scene collaboration for robust multimodal perception in autonomous driving.

Abstract

Bird's eye view (BEV) representation has emerged as a dominant solution for describing 3D space in autonomous driving scenarios. However, objects in the BEV representation typically exhibit small sizes, and the associated point cloud context is inherently sparse, which leads to great challenges for reliable 3D perception. In this paper, we propose IS-Fusion, an innovative multimodal fusion framework that jointly captures the Instance- and Scene-level contextual information. IS-Fusion essentially differs from existing approaches that only focus on the BEV scene-level fusion by explicitly incorporating instance-level multimodal information, thus facilitating the instance-centric tasks like 3D object detection. It comprises a Hierarchical Scene Fusion (HSF) module and an Instance-Guided Fusion (IGF) module. HSF applies Point-to-Grid and Grid-to-Region transformers to capture the multimodal scene context at different granularities. IGF mines instance candidates, explores their relationships, and aggregates the local multimodal context for each instance. These instances then serve as guidance to enhance the scene feature and yield an instance-aware BEV representation. On the challenging nuScenes benchmark, IS-Fusion outperforms all the published multimodal works to date. Code is available at: https://github.com/yinjunbo/IS-Fusion.

Figures (6)

• Figure 1: Motivation of IS-Fusion. (a) Previous approaches typically focus on fusion at the entire scene level during multimodal encoding. (b) In contrast, IS-Fusion places additional emphasis on the fusion at the instance level and explores the instance-to-scene collaboration to enhance the overall representation.
• Figure 2: Overview of our IS-Fusion framework. Multimodal inputs including a point cloud and multi-view images are first processed by modality-specific encoders to obtain initial features. Then, the HSF module, equipped with Point-to-Grid and Grid-to-Region transformers, utilizes these features to generate a scene-level feature with hierarchical context. Furthermore, the IGF module identifies the top-$K$ salient instances and aggregates the multimodal context for each instance. Finally, these instances are employed by the Instance-to-Scene transformer to propagate valuable information to the scene, producing the final BEV representation with improved instance awareness.
• Figure 3: Illustration of HSF module. It first aggregates the point-level features into the grid-level features with the Point-to-Grid transformer, and then explores the inter-grid and inter-region feature interaction through the Grid-to-Region transformer.
• Figure 4: Illustration of IGF module. Instance candidates are first initialized based on the BVE heatmap. Then, we perform reasoning on these instances, meanwhile aggregating rich semantic context from the image features. Finally, these instances transfer contextual information to the BEV scene feature through an Instance-to-Scene transformer attention mechanism.
• Figure 5: Examples of 3D object detections on nuScenes validation set. We visualize the 3D bounding boxes of car, pedestrian and bicycle with orange, blue and red colors in the multi-view images. In the point cloud, the predictions are in gray and GTs are in green.