Pragmatic Heterogeneous Collaborative Perception via Generative Communication Mechanism

TL;DR

GenComm addresses pragmatic heterogeneous collaborative perception by enabling feature generation across diverse agents without altering core networks. It introduces a Deformable Message Extractor, a Spatial-Aware Feature Generator based on conditional diffusion, and a Channel Enhancer to fuse generated collaborator features with ego representations. The training adopts a two-stage scheme (homogeneous pre-training followed by lightweight extractor alignment) to preserve semantic consistency while enabling scalable inclusion of new agents, achieving substantial reductions in communication cost and parameter count. Experiments across OPV2V-H, DAIR-V2X, and V2X-Real demonstrate superior accuracy and robustness under pose errors and time delays, with strong scalability as agents join or drop out. Overall, GenComm offers a practical solution for real-world multi-agent perception with improved safety implications for autonomous driving.

Abstract

Multi-agent collaboration enhances the perception capabilities of individual agents through information sharing. However, in real-world applications, differences in sensors and models across heterogeneous agents inevitably lead to domain gaps during collaboration. Existing approaches based on adaptation and reconstruction fail to support pragmatic heterogeneous collaboration due to two key limitations: (1) Intrusive retraining of the encoder or core modules disrupts the established semantic consistency among agents; and (2) accommodating new agents incurs high computational costs, limiting scalability. To address these challenges, we present a novel Generative Communication mechanism (GenComm) that facilitates seamless perception across heterogeneous multi-agent systems through feature generation, without altering the original network, and employs lightweight numerical alignment of spatial information to efficiently integrate new agents at minimal cost. Specifically, a tailored Deformable Message Extractor is designed to extract spatial message for each collaborator, which is then transmitted in place of intermediate features. The Spatial-Aware Feature Generator, utilizing a conditional diffusion model, generates features aligned with the ego agent's semantic space while preserving the spatial information of the collaborators. These generated features are further refined by a Channel Enhancer before fusion. Experiments conducted on the OPV2V-H, DAIR-V2X and V2X-Real datasets demonstrate that GenComm outperforms existing state-of-the-art methods, achieving an 81% reduction in both computational cost and parameter count when incorporating new agents. Our code is available at https://github.com/jeffreychou777/GenComm.

Figures (10)

• Figure 1: Comparison of heterogeneous collaboration strategies. (a) Adaptation-based strategy: transform features via one or two-stage adaptation. (b) Reconstruction-based strategy: reconstruct features on ego agent using indices of a shared codebook. (c) Ours (Generation-based): generate features locally using collaborators' spatial messages, requiring only lightweight extractor tuning to support new agents without modifying the core module. (d) Our method shows superior scalability over (a) and (b).
• Figure 2: Framework overview of GenComm. The Deformable Message Extractor extracts spatial messages, which serve as conditions for the Spatial-Aware Feature Generator. A Channel Enhancer further refines the generated features before fusion.
• Figure 3: Comparision of performance with baselines on the pose error and time delay setting.
• Figure 4: Ablation study on the channel size of the spatial message.
• Figure 5: Visualization result of our generation method, Compared to CodeFilling, generated feature$\hat{\mathcal{F}_i}$ is more consistent with the ego agent $i$'s semantic space and preserving the spatial information from collaborator $j$. The prediction and GT shown in red and green respectively.