CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers

TL;DR

CoBEVT tackles the challenge of accurate BEV semantic segmentation in cooperative multi-agent camera systems. It introduces FAX, a 3D fused axial attention mechanism that combines local windowed and sparse global attention to fuse BEV features across multiple agents and camera views efficiently. The framework comprises SinBEVT for high-resolution single-agent BEV feature extraction and FuseBEVT for cross-agent fusion, achieving state-of-the-art performance on OPV2V and strong generalization to LiDAR-based 3D detection and nuScenes. The work demonstrates the practical impact of sparse 3D attention for scalable, real-time cooperative perception in autonomous driving, while noting the need for real-world validation under synchronization and weather variations.

Abstract

Bird's eye view (BEV) semantic segmentation plays a crucial role in spatial sensing for autonomous driving. Although recent literature has made significant progress on BEV map understanding, they are all based on single-agent camera-based systems. These solutions sometimes have difficulty handling occlusions or detecting distant objects in complex traffic scenes. Vehicle-to-Vehicle (V2V) communication technologies have enabled autonomous vehicles to share sensing information, dramatically improving the perception performance and range compared to single-agent systems. In this paper, we propose CoBEVT, the first generic multi-agent multi-camera perception framework that can cooperatively generate BEV map predictions. To efficiently fuse camera features from multi-view and multi-agent data in an underlying Transformer architecture, we design a fused axial attention module (FAX), which captures sparsely local and global spatial interactions across views and agents. The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation. Moreover, CoBEVT is shown to be generalizable to other tasks, including 1) BEV segmentation with single-agent multi-camera and 2) 3D object detection with multi-agent LiDAR systems, achieving state-of-the-art performance with real-time inference speed. The code is available at https://github.com/DerrickXuNu/CoBEVT.

Figures (11)

• Figure 1: The overall framework of CoBEVT. White boxes in prediction maps indicate car segmentation results.
• Figure 2: Illustrated examples of fused axial attention (FAX) in two use cases -- (a) multi-agent BEV fusion and (b) multi-view camera fusion. FAX attends to 3D local windows (red) and sparse global tokens (blue) to attain location-wise and contextual-aware aggregation. In (b), for example, the white van is torn apart in three views (front-right, back, and back-left), our sparse global attention can capture long-distance relationships across parts in different views to attain global contextual understanding.
• Figure 3: Architectures of (a) SinBEVT and FuseBEVT, and (b) the FAX-SA and FAX-CA block.
• Figure 4: Qualitative results of CoBEVT. From left to right: the front camera image of (a) ego, (b) av1, (c) av2, (d) groundtruth and (e) prediction. The green bounding boxes represent ego vehicles, while the white boxes denote the segmented vehicles. CoBEVT demonstrates robust performance under various traffic situations and road types. It is also capable of detecting occluded or distant vehicles (white circled) benefiting from the collaboration.
• Figure 5: Ablation studies. (a) IoU vs. number of dropped cameras (b) IoU vs. number of agents. (c) FPS vs. number of agents. The channel dimension of BEV feature map is fixed as 128 for (c).