RESAR-BEV: An Explainable Progressive Residual Autoregressive Approach for Camera-Radar Fusion in BEV Segmentation

TL;DR

RESAR-BEV tackles BEV semantic segmentation for autonomous driving under sensor misalignment and noise by reframing the task as progressive residual refinement. It introduces a dual-branch camera-radar encoding pipeline and a cascaded Drive-Transformer/Modifier-Transformer that perform coarse-to-fine BEV refinement via GT token-map decomposition and residual fusion. The method employs ground-proximity lifting, height-aware attention, and multi-scale supervision with a BEV-GT decoder and adaptive Dice loss, achieving $54.0\%$ mIoU on nuScenes at $14.6$ FPS. This approach enhances robustness to long-range perception and adverse weather while maintaining real-time efficiency, offering interpretable insights through staged residuals and cross-modal attention patterns.

Abstract

Bird's-Eye-View (BEV) semantic segmentation provides comprehensive environmental perception for autonomous driving but suffers multi-modal misalignment and sensor noise. We propose RESAR-BEV, a progressive refinement framework that advances beyond single-step end-to-end approaches: (1) progressive refinement through residual autoregressive learning that decomposes BEV segmentation into interpretable coarse-to-fine stages via our Drive-Transformer and Modifier-Transformer residual prediction cascaded architecture, (2) robust BEV representation combining ground-proximity voxels with adaptive height offsets and dual-path voxel feature encoding (max+attention pooling) for efficient feature extraction, and (3) decoupled supervision with offline Ground Truth decomposition and online joint optimization to prevent overfitting while ensuring structural coherence. Experiments on nuScenes demonstrate RESAR-BEV achieves state-of-the-art performance with 54.0% mIoU across 7 essential driving-scene categories while maintaining real-time capability at 14.6 FPS. The framework exhibits robustness in challenging scenarios of long-range perception and adverse weather conditions.

Figures (8)

• Figure 1: (a) Previous single-step end-to-end multimodal BEV scene segmentation approach; (b) Our proposed progressive autoregressive residual prediction method with multi-resolution Ground Truth and multi-stage loss supervision; (c) Visualization of the RAF module's multi-scale feature generation and accumulation process through decoder mapping to BEV semantics, followed by upsampling to uniform resolution.
• Figure 2: Our Progressive Residual-Autoregressive BEV Segmentation Framework: (a) GT-Encoder-Decoder decomposes GroundTruth into multi-resolution residuals for hierarchical supervision; (b) Dual-branch encoder processes camera and radar inputs. The RAF module's Driver-T generates low-resolution BEV via cross-modal attention (with dynamic Height-Offset/Compress for ground features), followed by Modifier-T predicting multi-scale residuals through autoregressive refinement (integrating historical outputs and radar features via resolution/channel-wise gates); (c) Each stage computes residual difference losses via BEV-GT-Decoder, with the final output generating segmentation maps and segmentation loss.
• Figure 3: Voxel feature extraction: we normalize each voxel to $10$ points, then extract $C\times 10$ features via point-wise encoding. Apply parallel max/attention-pooling, concatenate with original features $(3C\times 10)$, and compress to C channels via MLP. Repeat twice, then max-pool for final voxel features.
• Figure 4: Lifting and Unlifting Visualization Based on Camera Sensor Intrinsics.
• Figure 5: Architecture of Driver/Modifier Transformer decoders. Cascaded decoders process learnable 3-layer $f_{bev}$, where Cross Deformable Attention enables BEV-to-multi-view semantic interaction. Modifier stages maintain independent cross-attention modules while sharing other components.