More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models

TL;DR

This work tackles whether reasoning produced by vision-language driving agents causally informs planning. It introduces DriveMind, a nuPlan-based dataset with plan-aligned Chain-of-Thought (CoT) and modular priors, enabling controlled ablations and causal diagnostics. Across multiple architectures, experiments reveal a persistent reasoning–planning disconnect: planning relies primarily on textual priors and is only weakly influenced by CoT, with GRPO failing to restore a causal link. The authors propose a training-free causal probe and sequence-level attention analysis to diagnose shortcut learning, and they outline concrete directions (contrastive pre-finetuning and negative-example learning) to cultivate causally faithful planning. Overall, DriveMind provides a valuable dataset and diagnostic toolkit for evaluating and improving causal fidelity in future VLM driving models.

Abstract

Vision-Language Model (VLM) driving agents promise explainable end-to-end autonomy by first producing natural-language reasoning and then predicting trajectory planning. However, whether planning is causally driven by this reasoning remains a critical but unverified assumption. To investigate this, we build DriveMind, a large-scale driving Visual Question Answering (VQA) corpus with plan-aligned Chain-of-Thought (CoT), automatically generated from nuPlan. Our data generation process converts sensors and annotations into structured inputs and, crucially, separates priors from to-be-reasoned signals, enabling clean information ablations. Using DriveMind, we train representative VLM agents with Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) and evaluate them with nuPlan's metrics. Our results, unfortunately, indicate a consistent causal disconnect in reasoning-planning: removing ego/navigation priors causes large drops in planning scores, whereas removing CoT produces only minor changes. Attention analysis further shows that planning primarily focuses on priors rather than the CoT. Based on this evidence, we propose the Reasoning-Planning Decoupling Hypothesis, positing that the training-yielded reasoning is an ancillary byproduct rather than a causal mediator. To enable efficient diagnosis, we also introduce a novel, training-free probe that measures an agent's reliance on priors by evaluating its planning robustness against minor input perturbations. In summary, we provide the community with a new dataset and a diagnostic tool to evaluate the causal fidelity of future models.

Figures (6)

• Figure 1: An overview of our framework for investigating the reasoning-planning disconnect in VLM agents. (Left) The DriveMind dataset generation pipeline. (Middle) Our methodology for causal validation. (Right) An illustration of our core finding, where planning follows a textual prior shortcut, bypassing the correct CoT reasoning.
• Figure 2: An illustration of our causal probe's diagnostic capabilities. The figure contrasts an agent's original plan with its divergent trajectories when subjected to two types of perturbations (lateral offset and direction inversion). Aggregated statistics quantify the failure modes, and the highlighted inset shows a direct contradiction between the agent's reasoning and its final plan.
• Figure 3: An illustration of our causal probe's diagnostic capabilities in a straight driving scenario. The figure contrasts an agent's original plan with its divergent trajectories when subjected to lateral offset perturbations to the left and right. The lateral trajectories of both models' output undergo a significant change in response to the perturbations.
• Figure 4: An illustration of our causal probe's diagnostic capabilities in a stopping scenario. The figure contrasts an agent's original plan with its divergent trajectories when subjected to lateral offset perturbations to the left and right. Models output distinct lateral ‘translation’ trajectories after perturbation, fully demonstrating the disconnect between reasoning and planning, as well as the existence of shortcut learning dependent on text priors.
• Figure 5: Prompt for GPT-4.1 to generate analysis for different parts. All together combine to form the plan-aligned CoT in DriveMind.