Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VILTA: A VLM-in-the-Loop Adversary for Enhancing Driving Policy Robustness

Qimao Chen, Fang Li, Shaoqing Xu, Zhiyi Lai, Zixun Xie, Yuechen Luo, Shengyin Jiang, Hanbing Li, Long Chen, Bing Wang, Yi Zhang, Zhi-Xin Yang

TL;DR

This work tackles the long-tail problem in autonomous driving by introducing VILTA, a VLM-in-the-loop adversary that directly edits surrounding agents' future trajectories to generate diverse, safety-critical training scenarios. By integrating a Vision-Language Model into the closed-loop RL framework (VLE paradigm), VILTA produces plausible yet challenging trajectories grounded in BEV scene representations and map information, followed by post-processing with B-spline smoothing and LQR control. Across CARLA and nuScenes experiments, VILTA achieves higher route completion and lower collision rates in challenging scenarios while largely preserving performance on normal scenarios, indicating strong robustness to long-tail events and resilience against catastrophic forgetting. The approach demonstrates the value of directly leveraging VLM generative capabilities within the training loop to craft a dynamic curriculum of adversarial interactions, with potential for broader applicability in safety-critical robotics and end-to-end autonomous driving paradigms, albeit with sim-to-real validation as a future step.

Abstract

The safe deployment of autonomous driving (AD) systems is fundamentally hindered by the long-tail problem, where rare yet critical driving scenarios are severely underrepresented in real-world data. Existing solutions including safety-critical scenario generation and closed-loop learning often rely on rule-based heuristics, resampling methods and generative models learned from offline datasets, limiting their ability to produce diverse and novel challenges. While recent works leverage Vision Language Models (VLMs) to produce scene descriptions that guide a separate, downstream model in generating hazardous trajectories for agents, such two-stage framework constrains the generative potential of VLMs, as the diversity of the final trajectories is ultimately limited by the generalization ceiling of the downstream algorithm. To overcome these limitations, we introduce VILTA (VLM-In-the-Loop Trajectory Adversary), a novel framework that integrates a VLM into the closed-loop training of AD agents. Unlike prior works, VILTA actively participates in the training loop by comprehending the dynamic driving environment and strategically generating challenging scenarios through direct, fine-grained editing of surrounding agents' future trajectories. This direct-editing approach fully leverages the VLM's powerful generalization capabilities to create a diverse curriculum of plausible yet challenging scenarios that extend beyond the scope of traditional methods. We demonstrate that our approach substantially enhances the safety and robustness of the resulting AD policy, particularly in its ability to navigate critical long-tail events.

VILTA: A VLM-in-the-Loop Adversary for Enhancing Driving Policy Robustness

TL;DR

This work tackles the long-tail problem in autonomous driving by introducing VILTA, a VLM-in-the-loop adversary that directly edits surrounding agents' future trajectories to generate diverse, safety-critical training scenarios. By integrating a Vision-Language Model into the closed-loop RL framework (VLE paradigm), VILTA produces plausible yet challenging trajectories grounded in BEV scene representations and map information, followed by post-processing with B-spline smoothing and LQR control. Across CARLA and nuScenes experiments, VILTA achieves higher route completion and lower collision rates in challenging scenarios while largely preserving performance on normal scenarios, indicating strong robustness to long-tail events and resilience against catastrophic forgetting. The approach demonstrates the value of directly leveraging VLM generative capabilities within the training loop to craft a dynamic curriculum of adversarial interactions, with potential for broader applicability in safety-critical robotics and end-to-end autonomous driving paradigms, albeit with sim-to-real validation as a future step.

Abstract

The safe deployment of autonomous driving (AD) systems is fundamentally hindered by the long-tail problem, where rare yet critical driving scenarios are severely underrepresented in real-world data. Existing solutions including safety-critical scenario generation and closed-loop learning often rely on rule-based heuristics, resampling methods and generative models learned from offline datasets, limiting their ability to produce diverse and novel challenges. While recent works leverage Vision Language Models (VLMs) to produce scene descriptions that guide a separate, downstream model in generating hazardous trajectories for agents, such two-stage framework constrains the generative potential of VLMs, as the diversity of the final trajectories is ultimately limited by the generalization ceiling of the downstream algorithm. To overcome these limitations, we introduce VILTA (VLM-In-the-Loop Trajectory Adversary), a novel framework that integrates a VLM into the closed-loop training of AD agents. Unlike prior works, VILTA actively participates in the training loop by comprehending the dynamic driving environment and strategically generating challenging scenarios through direct, fine-grained editing of surrounding agents' future trajectories. This direct-editing approach fully leverages the VLM's powerful generalization capabilities to create a diverse curriculum of plausible yet challenging scenarios that extend beyond the scope of traditional methods. We demonstrate that our approach substantially enhances the safety and robustness of the resulting AD policy, particularly in its ability to navigate critical long-tail events.
Paper Structure (34 sections, 3 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 34 sections, 3 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Safety-Critical Scenario Generation
  4. Closed-loop Learning for Autonomous Driving
  5. Preliminaries
  6. Partially Observable Markov Decision Process
  7. Problem Formulation
  8. Methodology
  9. VLM In-the-loop Trajectory Adversary
  10. Scene Representation
  11. Scene Understanding
  12. Trajectory Editing
  13. Trajectory Post-processing
  14. Experiments
  15. Experimental Setups
  16. ...and 19 more sections

Figures (5)

  • Figure 1: Comparison of different approaches to handling long-tail problem in autonomous driving. (a) Safety-critical scenario generation, which lacks use of the generated scenarios in training; (b) Closed-loop learning, which struggles to generate diverse scenarios; (c) Our proposed VLM-in-the-loop adversary, which is capable of generating scenarios that are both challenging and diverse.
  • Figure 2: Overview of VILTA framework. (a) Reinforcement learning environment executes the edited trajectory of the risky agent, facilitates the training of the ego agent, and provides the initial scene representation; (b) The raw scene data is processed into a specific representation formatted for input to VLM; (c) Gemini performs both scene understanding and trajectory editing in a single pass, outputting a raw edited trajectory; (d) The post-processing stage ensures that the final trajectory is both smooth and kinematically feasible.
  • Figure 3: Trajectory visualization and empirical analysis. Panels (a-f) provide trajectory visualizations; panels (g-i) visualize the diversity of the trajectory distributions; and panels (j-l) present box plots of key trajectory features.
  • Figure 4: A complete sample of the output generated by the VLM.
  • Figure 5: Edited trajectories visualization in both nuScenes and CARLA environments.