Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning

TL;DR

Counterfactual VLA (CF-VLA) addresses the lack of internal self-critique in reasoning-augmented Vision-Language-Action systems for autonomous driving by enabling counterfactual reasoning over its own predicted meta-actions. It introduces time-segmented meta-actions, a rollout-filter-label pipeline to mine high-value failure cases, and a counterfactual reasoning step that revises plans before trajectory generation, yielding adaptive thinking that concentrates effort on harder scenarios. Empirical results show substantial gains in trajectory accuracy (up to 17.6%) and safety (up to 20.5%), with multi-round CF data further enhancing performance while reducing think rate. The work demonstrates that integrating self-reflection into embodied agents can produce safer, more reliable driving policies and offers a general paradigm for self-improving action-conditioned reasoning in AI systems.

Abstract

Recent reasoning-augmented Vision-Language-Action (VLA) models have improved the interpretability of end-to-end autonomous driving by generating intermediate reasoning traces. Yet these models primarily describe what they perceive and intend to do, rarely questioning whether their planned actions are safe or appropriate. This work introduces Counterfactual VLA (CF-VLA), a self-reflective VLA framework that enables the model to reason about and revise its planned actions before execution. CF-VLA first generates time-segmented meta-actions that summarize driving intent, and then performs counterfactual reasoning conditioned on both the meta-actions and the visual context. This step simulates potential outcomes, identifies unsafe behaviors, and outputs corrected meta-actions that guide the final trajectory generation. To efficiently obtain such self-reflective capabilities, we propose a rollout-filter-label pipeline that mines high-value scenes from a base (non-counterfactual) VLA's rollouts and labels counterfactual reasoning traces for subsequent training rounds. Experiments on large-scale driving datasets show that CF-VLA improves trajectory accuracy by up to 17.6%, enhances safety metrics by 20.5%, and exhibits adaptive thinking: it only enables counterfactual reasoning in challenging scenarios. By transforming reasoning traces from one-shot descriptions to causal self-correction signals, CF-VLA takes a step toward self-reflective autonomous driving agents that learn to think before they act.

Figures (16)

• Figure 1: Counterfactual Vision-Language-Action (CF-VLA) Model. Top: CF-VLA conducts reasoning adaptively. The model engages in reasoning more frequently and achieves more significant task performance gains in complex scenarios that have higher trajectory errors. Bottom: CF-VLA reflects on its own action plan and corrects it before generating the final trajectory.
• Figure 2: The framework of CF-VLA. A base VLA is fine-tuned on a counterfactual reasoning dataset generated by a rollout–filter–label pipeline. The resulting CF-VLA supports both direct inference and self-reflective inference, in which counterfactual reasoning edits meta-actions before trajectory generation.
• Figure 3: (A) Adaptive Reasoning can be achieved by training models on a mixture of data with the unified instruction prompt. (B) Data generation process. We build a rollout–filter–label pipeline that runs the VLA, detects samples where its meta-actions are problematic, and labels counterfactual (CF) reasoning traces, forming a CF reasoning dataset. (C) Data filtering process. We use trajectory disagreement between trajectories that are free-generated and those induced by the ground-truth meta-actions to filter data. Each data point is colored by the meta-actions IOU in free generation.
• Figure 4: The dataset composition. We use a subset of the meta-action-labeled dataset $\mathcal{D}_\text{meta}$ as the validation set $\mathcal{D}_\text{val}$.
• Figure 5: Qualitative results of CF-VLA. For three representative and safety-critical scenarios, each row shows the model's initial meta-actions (left), the reasoning trace (middle), and the updated meta-actions (right) together with the resulting trajectory. The counterfactual reasoning step identifies issues (missing lane changes, late turns, and failure to slow for pedestrians) and edits the meta-actions accordingly.