Traffic Regulation-aware Path Planning with Regulation Databases and Vision-Language Models

TL;DR

This work tackles the challenge of ensuring ADS comply with traffic regulations across jurisdictions by introducing a regulation-aware path-planning framework that combines a machine-readable regulation database, a finite-state machine (FSM), and a vision-language model (VLM). The framework uses mission and motion planning to evaluate candidate plans against regulatory constraints, with a cost function balancing legality, safety, comfort, and progress, and employs a VLM (LLaVA) to produce scene-based textual descriptions that inform legality assessments. It is validated through simulation in UCLA OpenCDA and real-world UCLA campus tests, demonstrating the system’s ability to handle complex multi-regulation scenarios and to operate at real-time inference speeds (noting current VLM speed limitations). The results indicate the VLM’s potential to aid high-level perception and reasoning in regulation-aware driving while also highlighting needs for domain-specific tuning and improved sign recognition to enhance reliability in diverse traffic environments.

Abstract

This paper introduces and tests a framework integrating traffic regulation compliance into automated driving systems (ADS). The framework enables ADS to follow traffic laws and make informed decisions based on the driving environment. Using RGB camera inputs and a vision-language model (VLM), the system generates descriptive text to support a regulation-aware decision-making process, ensuring legal and safe driving practices. This information is combined with a machine-readable ADS regulation database to guide future driving plans within legal constraints. Key features include: 1) a regulation database supporting ADS decision-making, 2) an automated process using sensor input for regulation-aware path planning, and 3) validation in both simulated and real-world environments. Particularly, the real-world vehicle tests not only assess the framework's performance but also evaluate the potential and challenges of VLMs to solve complex driving problems by integrating detection, reasoning, and planning. This work enhances the legality, safety, and public trust in ADS, representing a significant step forward in the field.

Figures (7)

• Figure 1: The overview of the path planning framework where the path planning modules are highlighted in blue, the interaction between VLM and mission planning is highlighted in orange, and the ADS regulation database is highlighted in green.
• Figure 2: FSM superstates and transitions overview. Each superstate contains additional states and corresponding transitions.
• Figure 3: Framework structure of the VLM integrated path planning framework.
• Figure 4: ADS vehicle and the real-time WebUI interface for VLM and FSM visualization.
• Figure 5: Trajectory plot and the simulation screenshot when ego ADS vehicle overtaking a cyclist.