V3LMA: Visual 3D-enhanced Language Model for Autonomous Driving

TL;DR

V3LMA tackles the challenge of 3D scene understanding in autonomous driving by fusing textual descriptions derived from 3D detections with video-based reasoning from LVLMs in a zero-shot framework. A modular 3D understanding pipeline (object detection, tracking, and monocular depth) feeds textual cues into a cross-model fusion with a large language model, avoiding fine-tuning. Through extensive ablations over head weights, feature merging, fusion depth, and prompt design, it achieves competitive LingoQA performance (0.56 on the full dataset and 0.60 on a subset) without domain-specific training, highlighting the practical viability of zero-shot 3D-aware multimodal reasoning for safer autonomous driving. The work further provides actionable guidance on fusion strategies and prompt construction, and demonstrates the potential for generalization to other vision-language tasks in driving contexts.

Abstract

Large Vision Language Models (LVLMs) have shown strong capabilities in understanding and analyzing visual scenes across various domains. However, in the context of autonomous driving, their limited comprehension of 3D environments restricts their effectiveness in achieving a complete and safe understanding of dynamic surroundings. To address this, we introduce V3LMA, a novel approach that enhances 3D scene understanding by integrating Large Language Models (LLMs) with LVLMs. V3LMA leverages textual descriptions generated from object detections and video inputs, significantly boosting performance without requiring fine-tuning. Through a dedicated preprocessing pipeline that extracts 3D object data, our method improves situational awareness and decision-making in complex traffic scenarios, achieving a score of 0.56 on the LingoQA benchmark. We further explore different fusion strategies and token combinations with the goal of advancing the interpretation of traffic scenes, ultimately enabling safer autonomous driving systems.

Figures (12)

• Figure 1: 3D information is essential for accurate and safe scene understanding in autonomous driving. While can extract rich semantic information from images, they struggle to capture 3D spatial cues. To address this limitation, we propose enhancing visual scene understanding by incorporating textual descriptions of 3D detections. These descriptions are processed by an , which is better equipped to handle textual input than an .
• Figure 2: Overview of the approach of this work. At the left side a pre-trained base LLM is getting a textual description of surrounding objects and a question as input. On the right side, a pre-trained LVLM with the same base LLM as on the left-hand side uses the video and the question as input to reason about the visual content of the scene. The information of both models is combined to enhance the overall scene understanding.
• Figure 3: Pipeline for obtaining locations and descriptions for traffic signs, trafic lights and moving objects in the scene.
• Figure 4: Two different strategies to process the combined features. On the left, the LVLM receives only its own features from the previous layers (False). The LLM processes the combined input of both models. On the right, both models receive the combined features of the previous layer as input (True).
• Figure 5: Example for traffic light detection and state recognition using the fine-tuned YOLO model provided by KASTEL-MobilityLab’s. The model is capable of detecting direction specific traffic lights.