Robust Driving QA through Metadata-Grounded Context and Task-Specific Prompts

TL;DR

This work tackles high-level driving QA by pairing a two-phase framework with domain-grounded context. Phase-1 leverages six-view imagery, short history, chain-of-thought prompts, and self-consistency to improve reasoning, while Phase-2 injects nuScenes metadata and category-specific prompts to further ground the model's answers. Key findings show substantial accuracy gains, with Phase-2 achieving 67.37% overall and the system maintaining ~96% accuracy under severe visual corruption. The results demonstrate that careful prompt design and structured grounding can significantly enhance vision-language models for interpretable, safe autonomous-driving decision support. The approach offers a scalable roadmap for integrating rich scene context into VLM-based driving QA systems, with clear avenues for future improvements such as richer scene understanding and uncertainty estimation.

Abstract

We present a two-phase vision-language QA system for autonomous driving that answers high-level perception, prediction, and planning questions. In Phase-1, a large multimodal LLM (Qwen2.5-VL-32B) is conditioned on six-camera inputs, a short temporal window of history, and a chain-of-thought prompt with few-shot exemplars. A self-consistency ensemble (multiple sampled reasoning chains) further improves answer reliability. In Phase-2, we augment the prompt with nuScenes scene metadata (object annotations, ego-vehicle state, etc.) and category-specific question instructions (separate prompts for perception, prediction, planning tasks). In experiments on a driving QA benchmark, our approach significantly outperforms the baseline Qwen2.5 models. For example, using 5 history frames and 10-shot prompting in Phase-1 yields 65.1% overall accuracy (vs.62.61% with zero-shot); applying self-consistency raises this to 66.85%. Phase-2 achieves 67.37% overall. Notably, the system maintains 96% accuracy under severe visual corruption. These results demonstrate that carefully engineered prompts and contextual grounding can greatly enhance high-level driving QA with pretrained vision-language models.

Figures (13)

• Figure 1: Phase-1 overview. Multi-view inputs (6-CAM) and a short temporal window of history frames are fused with a few-shot QA (DriveBench QA) into a domain-aware Prompt, which conditions Qwen 2.5-VL 32B.
• Figure 2: Phase-2 overview. A rule-based router splits the input question by category. NuScenes metadata (pose/map), zoom-in crops (question target), and object/ego status (text) are injected to form a task-specific prompt, then fed to Qwen 2.5-VL 32B.
• Figure 3: Visual prompts for the vanishing point on clean image. The yellow line represents the horizontal line, and the estimated vanishing point is marked with 'VP'.
• Figure 4: Visual prompts for the vanishing point on corrupted image. The yellow line represents the horizontal line, and the estimated vanishing point is marked with 'VP'.
• Figure 5: Visual prompts for the dominant gradient orientation on a clean image. Brighter regions indicate stronger gradients, darker regions indicate weaker gradients, and colors represent gradient directions.