BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

TL;DR

This work tackles BEV semantic segmentation with arbitrary camera configurations by introducing BEVSegFormer, a transformer-based pipeline that uses a shared backbone, a deformable Transformer Encoder, and a BEV Transformer Decoder equipped with a MultiCameraDeformAttn module to perform BEV-to-image view transformation without camera extrinsics. The approach yields state-of-the-art BEV segmentation on the nuScenes dataset without temporal information and demonstrates robustness across single-camera and multi-camera setups. Ablation studies verify the contributions of the deformable cross-attention, the encoder/decoder design, and the absence of explicit camera parameters. The method holds practical significance for autonomous driving by enabling flexible, parameter-free BEV sensing across diverse sensor rigs.

Abstract

Semantic segmentation in bird's eye view (BEV) is an important task for autonomous driving. Though this task has attracted a large amount of research efforts, it is still challenging to flexibly cope with arbitrary (single or multiple) camera sensors equipped on the autonomous vehicle. In this paper, we present BEVSegFormer, an effective transformer-based method for BEV semantic segmentation from arbitrary camera rigs. Specifically, our method first encodes image features from arbitrary cameras with a shared backbone. These image features are then enhanced by a deformable transformer-based encoder. Moreover, we introduce a BEV transformer decoder module to parse BEV semantic segmentation results. An efficient multi-camera deformable attention unit is designed to carry out the BEV-to-image view transformation. Finally, the queries are reshaped according the layout of grids in the BEV, and upsampled to produce the semantic segmentation result in a supervised manner. We evaluate the proposed algorithm on the public nuScenes dataset and a self-collected dataset. Experimental results show that our method achieves promising performance on BEV semantic segmentation from arbitrary camera rigs. We also demonstrate the effectiveness of each component via ablation study.

Figures (11)

• Figure 1: Comparison of lane segmentation results on image space and BEV space. (a) lane segmentation on image space, (b) BEV segmentation by IPM view transformation of (a), (c) our BEV lane segmentation.
• Figure 2: Comparison of view transformation between our BEV-to-image scheme and 3D-to-2D method in DETR3D wang2021detr3d.
• Figure 3: Overview of the proposed network. Our method consists of a Shared Backbone, a Transformer Encoder and a BEV Transformer Decoder module.
• Figure 4: Illustration of the Multi-Camera Deformable Cross-Attention module in BEV Transformer Decoder.
• Figure 5: Illustration of Semantic Decoder.