MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation

TL;DR

MaskBEV presents a unified, masked-attention based multi-task learning framework that jointly performs 3D object detection and BEV map segmentation within a single Transformer decoder. By introducing a task-agnostic decoder, spatial modulation for adaptive mask coverage, and scene-level feature aggregation (MWWA and ASPP), it exploits cross-task complementarities in BEV space. The approach yields state-of-the-art multitask performance on nuScenes with 72.9 NDS and 73.9 mIoU, while maintaining competitive latency and robustness across sensor configurations. This work demonstrates the viability of a unified BEV-based MTL paradigm, reducing need for task-specific heads and enabling more efficient joint perception for autonomous driving.

Abstract

Accurate and robust multimodal multi-task perception is crucial for modern autonomous driving systems. However, current multimodal perception research follows independent paradigms designed for specific perception tasks, leading to a lack of complementary learning among tasks and decreased performance in multi-task learning (MTL) due to joint training. In this paper, we propose MaskBEV, a masked attention-based MTL paradigm that unifies 3D object detection and bird's eye view (BEV) map segmentation. MaskBEV introduces a task-agnostic Transformer decoder to process these diverse tasks, enabling MTL to be completed in a unified decoder without requiring additional design of specific task heads. To fully exploit the complementary information between BEV map segmentation and 3D object detection tasks in BEV space, we propose spatial modulation and scene-level context aggregation strategies. These strategies consider the inherent dependencies between BEV segmentation and 3D detection, naturally boosting MTL performance. Extensive experiments on nuScenes dataset show that compared with previous state-of-the-art MTL methods, MaskBEV achieves 1.3 NDS improvement in 3D object detection and 2.7 mIoU improvement in BEV map segmentation, while also demonstrating slightly leading inference speed.

Figures (6)

• Figure 1: Comparison between the multi-head multi-task perception framework with separated BEV encoder and our proposed MaskBEV. (a) Multiple task heads implement multi-task learning (MTL). The previous methods bevfusion_mitge2023metabevwang2023unitr adopt independent task head design. (b) The unified multi-task head design fully exploits the complementary advantages between multiple tasks, and uses one decoder to perform MTL in the unified BEV features.
• Figure 2: Overview of our MaskBEV framework. Multimodal input is passed through the feature encoding network to obtain the fused BEV features. Based on unified BEV features, our MaskBEV performs BEV map segmentation and 3D detection tasks on a unified Transformer decoder. Multi-task perception is not a simple task stacking, but a composite task learning process that promotes each other by utilizing the complementary characteristics of tasks.
• Figure 3: Illustration of attention mask. Left, purple represents the modulated mask, and we superimpose the ground truth. Right, yellow represents modulated 3D objects and green boxes represent ground truth. The mask of an object whose center point is on the segmentation mask is not drawn.
• Figure 4: Illustration of scene-level feature aggregation. In MWWA, multi-attention heads independently calculate attention in windows of different sizes to capture multi-scale features. ASPP captures the scene-level semantic layout of BEV features.
• Figure 5: Qualitative results of MaskBEV on MTL, including 3D object detection and BEV map segmentation tasks. MaskBEV shows better results than the MTL variant of UniTR wang2023unitr and comparable results to the STL variant of UniTR.