CollaMamba: Efficient Collaborative Perception with Cross-Agent Spatial-Temporal State Space Model

TL;DR

This work pioneers the exploration of the Mamba's potential in collaborative perception by constructing a foundational backbone network based on spatial SSM and devise a history-aware feature boosting module based on temporal SSM, which outperforms state-of-the-art methods.

Abstract

By sharing complementary perceptual information, multi-agent collaborative perception fosters a deeper understanding of the environment. Recent studies on collaborative perception mostly utilize CNNs or Transformers to learn feature representation and fusion in the spatial dimension, which struggle to handle long-range spatial-temporal features under limited computing and communication resources. Holistically modeling the dependencies over extensive spatial areas and extended temporal frames is crucial to enhancing feature quality. To this end, we propose a resource efficient cross-agent spatial-temporal collaborative state space model (SSM), named CollaMamba. Initially, we construct a foundational backbone network based on spatial SSM. This backbone adeptly captures positional causal dependencies from both single-agent and cross-agent views, yielding compact and comprehensive intermediate features while maintaining linear complexity. Furthermore, we devise a history-aware feature boosting module based on temporal SSM, extracting contextual cues from extended historical frames to refine vague features while preserving low overhead. Extensive experiments across several datasets demonstrate that CollaMamba outperforms state-of-the-art methods, achieving higher model accuracy while reducing computational and communication overhead by up to 71.9% and 1/64, respectively. This work pioneers the exploration of the Mamba's potential in collaborative perception. The source code will be made available.

Figures (8)

• Figure 1: The framework of CollaMamba. (a) The structure of Cross-Agent Spatial Collaboration backbone network, yielding compact and comprehensive intermediate features; (b) Single-Agent History-Aware Feature Boosting module, extracting temporal contextual cues from local observation trajectory; (c) Cross-Agent Collaborative Prediction module, leveraging the broad spatial-temporal features extracted from the Global Feature Trajectory to predict missing information from neighbor agents.
• Figure 2: The structure of Mamba Encoder and Decoder module. (a) The structure of Mamba Encoder; (b) The structure of Mamba Decoder, for reconstructing spatial features, the scale and dimensions are adjusted after the Mamba2D blocks to accommodate the detection head; (c) Mamba2D Block, 4-direction SSM scan is employed to better extracting the compact sequence-form spatial casual dependencies.
• Figure 3: The structure of Cross-Agent Fusion module.
• Figure 4: The structure of Historical Trajectory Encoder module.
• Figure 5: The structure of Single-Agent Feature Boosting module.