Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Yun Li, Yidu Zhang, Simon Thompson, Ehsan Javanmardi, Manabu Tsukada

Abstract

Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems.

Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Abstract

Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems.

Paper Structure

This paper contains 40 sections, 11 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. VLA Models and Text Structure for Driving
  4. Runtime Safety for Autonomous Driving
  5. Training-Time Safety Alignment
  6. Theoretical Foundations
  7. Text Structure as a Performance Bottleneck
  8. Three Principles of CSN
  9. Empirical Decomposition of Text Utility
  10. Taxonomy of Text Structure Approaches
  11. Methodology
  12. System Architecture Overview
  13. Causal Scene Narration Pipeline
  14. Environmental Data Extraction
  15. Causal Narration Generation
  16. ...and 25 more sections

Figures (6)

  • Figure 1: Three-stage framework overview. CSN (Stage 2) restructures driving-environment information into structured text, while the runtime safety supervisor (Stage 3) monitors VLA waypoints against $\mathcal{S}_{\text{sem}}$. Both modules operate on CPU.
  • Figure 2: The text structure bottleneck. (a) Template text presents intent $\mathcal{I}$ and constraints $\mathcal{C}$ as independent fragments with no $\mathcal{I} \to \mathcal{C}$ link. (b) CSN restores the causal link via explicit connectives (BUT, BEFORE).
  • Figure 3: CSN pipeline illustrated on a left-turn scenario. Each principle progressively transforms flat template text into quantitative, structured, and causally linked input.
  • Figure 4: Evaluation environments. (a) Town01, clear day. (b) Town03, heavy rain. (c) Town05, night. (d) Town07, dense fog.
  • Figure 5: Driving Score comparison across all ten configurations ($N{=}5$). Left and center: mean DS with 95% bootstrap CI. Right: raincloud plot showing KDE density (shaded), individual runs (dots), mean (diamond), and 95% CI (whisker). Dotted line: original LMDrive baseline.
  • ...and 1 more figures