MTA: Multimodal Task Alignment for BEV Perception and Captioning

TL;DR

MTA is introduced, a novel multimodal task alignment framework that boosts both BEV perception and captioning and underscores the effectiveness of unified alignment in reconciling BEV-based perception and captioning.

Abstract

Bird's eye view (BEV)-based 3D perception plays a crucial role in autonomous driving applications. The rise of large language models has spurred interest in BEV-based captioning to understand object behavior in the surrounding environment. However, existing approaches treat perception and captioning as separate tasks, focusing on the performance of only one task and overlooking the potential benefits of multimodal alignment. To bridge this gap between modalities, we introduce MTA, a novel multimodal task alignment framework that boosts both BEV perception and captioning. MTA consists of two key components: (1) BEV-Language Alignment (BLA), a contextual learning mechanism that aligns the BEV scene representations with ground-truth language representations, and (2) Detection-Captioning Alignment (DCA), a cross-modal prompting mechanism that aligns detection and captioning outputs. MTA seamlessly integrates into state-of-the-art baselines during training, adding no extra computational complexity at runtime. Extensive experiments on the nuScenes and TOD3Cap datasets show that MTA significantly outperforms state-of-the-art baselines in both tasks, achieving a 10.7% improvement in challenging rare perception scenarios and a 9.2% improvement in captioning. These results underscore the effectiveness of unified alignment in reconciling BEV-based perception and captioning.

Figures (2)

• Figure 1: Overview of proposed multimodal task alignment (MTA) framework. MTA enables joint performance improvement for BEV perception and dense captioning tasks through BEV-Language Alignment (BLA) and Detection-Captioning Alignment (DCA) mechanisms. In particular, BLA is a contextual learning mechanism to reconcile BEV scene representation with language-based scene understanding, and DCA is a cross-modal prompting mechanism to promote consistency between detection and captioning outputs. The MTA is trained end-to-end with a combination of task-specific losses (detection and captioning losses), along with the BLA and DCA objectives.
• Figure 2: Qualitative results comparing the proposed MTA with baseline methods on nuScenes and TOD3Cap datasets. Visualization results show that MTA shows improved alignment with ground-truth detections over the counterpart methods. Captioning results show that the proposed MTA generates captions that are more accurate in terms of the description and localization of objects over the TOD3Cap. Unlike MTA, TOD3Cap labels object 1 (a bus) as a trash can in the caption, illustrating a heightened risk of hallucination. $^{*}$We note that BEVFormer is only suited for perception tasks, thus caption is not provided.