CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model

TL;DR

This work addresses the challenge of transferring long-tail driving capabilities learned in simulation to real-world autonomous driving while preserving interpretability. It introduces CoC-VLA, a Chain-of-Causality Visual Language Model framework with a teacher VLM, a student VLM, and a transformer-based discriminator to align domains and enable end-to-end reasoning across temporal frames. The method uses a Text Adapter, Vision Adapter, LLM Brain, and CoC Answer to produce concise, causally linked driving instructions, and employs a novel backpropagation strategy to stabilize adversarial training. Experiments on CARLA-VLM and nuScenes-VLM demonstrate strong performance gains in language-based evaluation and open-loop driving metrics, validating the approach for robust sim-to-real transfer and explainable autonomous driving.

Abstract

Autonomous driving represents a prominent application of artificial intelligence. Recent approaches have shifted from focusing solely on common scenarios to addressing complex, long-tail situations such as subtle human behaviors, traffic accidents, and non-compliant driving patterns. Given the demonstrated capabilities of large language models (LLMs) in understanding visual and natural language inputs and following instructions, recent methods have integrated LLMs into autonomous driving systems to enhance reasoning, interpretability, and performance across diverse scenarios. However, existing methods typically rely either on real-world data, which is suitable for industrial deployment, or on simulation data tailored to rare or hard case scenarios. Few approaches effectively integrate the complementary advantages of both data sources. To address this limitation, we propose a novel VLM-guided, end-to-end adversarial transfer framework for autonomous driving that transfers long-tail handling capabilities from simulation to real-world deployment, named CoC-VLA. The framework comprises a teacher VLM model, a student VLM model, and a discriminator. Both the teacher and student VLM models utilize a shared base architecture, termed the Chain-of-Causality Visual-Language Model (CoC VLM), which integrates temporal information via an end-to-end text adapter. This architecture supports chain-of-thought reasoning to infer complex driving logic. The teacher and student VLM models are pre-trained separately on simulated and real-world datasets. The discriminator is trained adversarially to facilitate the transfer of long-tail handling capabilities from simulated to real-world environments by the student VLM model, using a novel backpropagation strategy.

Figures (3)

• Figure 1: Overview of the proposed framework. The framework comprises a teacher VLM model (based on the simulation CoC VLM baseline), a student VLM model (based on the real-world CoC VLM baseline), and a discriminator. The teacher and student VLM models share the same architecture but are trained separately on simulation and real-world datasets, respectively. The discriminator is employed to facilitate the adversarial transfer of capabilities from the simulation domain to the real-world domain. Train step1 refers to the adversarial training step1. Train step2 refers to the adversarial training step2.
• Figure 2: Qualitative results. Ego vehicle passing through a traffic light.
• Figure 3: Qualitative results. A scenario in which a car brakes in front of the ego vehicle.