mmCooper: A Multi-agent Multi-stage Communication-efficient and Collaboration-robust Cooperative Perception Framework

TL;DR

mmCooper addresses bandwidth constraints and calibration noise in cooperative perception by introducing an adaptive multi-stage fusion framework. It uses a Confidence-based Filter Generation to selectively transmit high-confidence late-stage bounding boxes and intermediate features, paired with a Multi-scale Offset-aware Fusion module and a Bounding Box Filtering & Calibration module to correct misalignments and refine detections. The approach yields substantial AP gains across OPV2V, DAIR-V2X, and V2XSet (e.g., AP@0.7 improvements of 7.29%, 1.31%, and 2.09%, respectively) while dramatically reducing communication volume (up to 18305x lower than some baselines). These results demonstrate the practicality of multi-stage collaboration for robust, bandwidth-efficient cooperative perception in real-world V2X scenarios.

Abstract

Collaborative perception significantly enhances individual vehicle perception performance through the exchange of sensory information among agents. However, real-world deployment faces challenges due to bandwidth constraints and inevitable calibration errors during information exchange. To address these issues, we propose mmCooper, a novel multi-agent, multi-stage, communication-efficient, and collaboration-robust cooperative perception framework. Our framework leverages a multi-stage collaboration strategy that dynamically and adaptively balances intermediate- and late-stage information to share among agents, enhancing perceptual performance while maintaining communication efficiency. To support robust collaboration despite potential misalignments and calibration errors, our framework prevents misleading low-confidence sensing information from transmission and refines the received detection results from collaborators to improve accuracy. The extensive evaluation results on both real-world and simulated datasets demonstrate the effectiveness of the mmCooper framework and its components.

Figures (15)

• Figure 1: Comparison between our proposed multi-stage mmCooper framework and existing methods. The BEV (Bird’s Eye View) communication map illustrates the information shared by agents at each scene location. (a)(b)(c) depict existing methods, which transmit the entire scene’s data in a single stage without considering sensing confidence, leading to excessive communication overhead and degraded performance. (d) demonstrates how mmCooper selectively transmits non-overlapping information across multiple stages, reducing communication costs and enhancing model performance.
• Figure 2: Overview of mmCooper framework: a cooperative perception system with adaptive multi-stage fusion. It consists of three main components: (1) Information Broadcasting (\ref{['subsec:broadcasting']}) filters features and bounding boxes to achieve bandwidth efficiency; (2) Intermediate-stage Fusion (\ref{['subsec:intermdiate fusion']}) captures surrounding information from the received feature maps for robust feature fusion; (3) Late-stage Fusion (\ref{['subsec:late']}) utilizes the information-rich fused features for Filtering & Calibration of the received bounding boxes.
• Figure 3: The design of Filter Generator in CFG Module, generating confidence scores to guide information transmission at each location.
• Figure 4: (a) The Multi-scale Offset-aware Fusion Module. (b) The Multi-scale Offset-aware Attention Module.
• Figure 5: (a) The BBox Filtering & Calibration (BFC) Module. (b) The Deformable BBox Attention (DBA) Module.