Work Zones challenge VLM Trajectory Planning: Toward Mitigation and Robust Autonomous Driving

TL;DR

The paper addresses the challenge of robust trajectory planning for autonomous driving in work zones, where irregular layouts and dynamic constraints degrade VLM performance. It conducts a systematic evaluation of multiple VLMs on the ROADWork dataset, revealing a high failure rate in work-zone scenarios and identifying eight failure patterns via scene-graph-based abnormal pattern analysis. To mitigate these failures, the authors propose REACT-Drive, a two-stage framework that offline builds a constraint-rule and executable-code database and online applies Retrieval-Augmented Generation (RAG) to guide trajectory generation using retrieved mitigation code. Empirical results show about a $3\times$ reduction in average displacement error (ADE) and sub-second inference times, with successful physical testing across 15 real-world work zones and zero collision rates in some cases, indicating strong practical potential for robust work-zone driving. Overall, the work demonstrates a concrete path to extend VLM capabilities into safety-critical, long-tail driving contexts by leveraging structured pattern analysis and retrieval-based guidance.

Abstract

Visual Language Models (VLMs), with powerful multimodal reasoning capabilities, are gradually integrated into autonomous driving by several automobile manufacturers to enhance planning capability in challenging environments. However, the trajectory planning capability of VLMs in work zones, which often include irregular layouts, temporary traffic control, and dynamically changing geometric structures, is still unexplored. To bridge this gap, we conduct the \textit{first} systematic study of VLMs for work zone trajectory planning, revealing that mainstream VLMs fail to generate correct trajectories in $68.0%$ of cases. To better understand these failures, we first identify candidate patterns via subgraph mining and clustering analysis, and then confirm the validity of $8$ common failure patterns through human verification. Building on these findings, we propose REACT-Drive, a trajectory planning framework that integrates VLMs with Retrieval-Augmented Generation (RAG). Specifically, REACT-Drive leverages VLMs to convert prior failure cases into constraint rules and executable trajectory planning code, while RAG retrieves similar patterns in new scenarios to guide trajectory generation. Experimental results on the ROADWork dataset show that REACT-Drive yields a reduction of around $3\times$ in average displacement error relative to VLM baselines under evaluation with Qwen2.5-VL. In addition, REACT-Drive yields the lowest inference time ($0.58$s) compared with other methods such as fine-tuning ($17.90$s). We further conduct experiments using a real vehicle in 15 work zone scenarios in the physical world, demonstrating the strong practicality of REACT-Drive.

Figures (5)

• Figure 1: REACT-Drive can effectively generate a correct planning trajectory in complex work zone scenarios. The red line denotes the QWEN2.5's output trajectory, the green line is GT, and the blue line denotes Real-REACT's output trajectory.
• Figure 2: Pipeline Overview. Abnormal Pattern Analysis builds scene graphs, clusters abnormal subgraphs, and summarizes patterns; $\mathsf{REACT\text{-}Drive}$ constructs a mitigation-code database from failure cases and retrieves the right code for new scenarios.
• Figure 3: Inference time comparison across different methods.
• Figure 4: Pattern coverage experiments.
• Figure A1: $8$ failure patterns (P1-P8) with per-pattern rule constraints.