Cross-view Transformers for real-time Map-view Semantic Segmentation

TL;DR

The paper tackles map-view semantic segmentation from multiple calibrated cameras without explicit depth estimation. It introduces cross-view transformers that use camera-aware positional encodings and cross-attention to implicitly align camera views with a shared map-view latent, achieving state-of-the-art results on nuScenes with real-time inference. Key contributions include a geometry-aware attention mechanism, a scalable two-scale architecture, and an end-to-end trainable framework that outperforms prior methods while offering faster inference. The approach demonstrates strong robustness to partial sensor dropout and provides qualitative evidence of learned geometric reasoning in attention patterns.

Abstract

We present cross-view transformers, an efficient attention-based model for map-view semantic segmentation from multiple cameras. Our architecture implicitly learns a mapping from individual camera views into a canonical map-view representation using a camera-aware cross-view attention mechanism. Each camera uses positional embeddings that depend on its intrinsic and extrinsic calibration. These embeddings allow a transformer to learn the mapping across different views without ever explicitly modeling it geometrically. The architecture consists of a convolutional image encoder for each view and cross-view transformer layers to infer a map-view semantic segmentation. Our model is simple, easily parallelizable, and runs in real-time. The presented architecture performs at state-of-the-art on the nuScenes dataset, with 4x faster inference speeds. Code is available at https://github.com/bradyz/cross_view_transformers.

Figures (6)

• Figure 1: We introduce an architecture for perception in a map-view frame from multiple views. Our model builds a map-view representation by cross-attending to image features. A camera-aware positional embedding can geometrically link up the camera and map-views.
• Figure 2: An overview of our proposed architecture for map-view segmentation. For each image, we extract image features across multiple scales. Using known camera pose and intrinsics, we construct a camera-aware positional embedding. We learn a map-view positional embedding that aggregates information from all views through a series of cross attention layers. Each cross-attention refines the map-view positional embedding and queries more accurate image locations.
• Figure 3: A comparison of model performance vs distance to the camera. Each entry shows the average intersection over union accuracy for annotations that are at least distance $d$ away.
• Figure 4: Degradation of our model as we randomly drop out $m \in \{0, 1, 2, 3\}$ cameras. The models performance drops linearly as the observed area shrinks roughly linearly with the number of cameras removed.
• Figure 5: Qualitative results on scenes with varying degrees of occlusion. Left shows the six camera views surrounding the vehicle. The top 3 views are front-facing, the bottom 3 views back-facing. On the right is our predicted map-view segmentation for vehicles and driveable area. Second from the right is the ground truth segmentation for reference. The ego-vehicle is located at the center of the map.