SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Models

TL;DR

SafeDrive addresses autonomous-vehicle safety in high-risk, long-tail scenarios by integrating an omnidirectional risk-quantification model with a knowledge- and data-driven, LLM-powered decision framework. It combines a Driver Risk Field (DRF) and Quantified Perceived Risk (QPR) with a GPT-4 driven agent and a memory-reflection loop to produce risk-aware actions that are continuously improved. Evaluations on HighD, InD, and RounD yield 100% safety across tested scenarios and human-decision alignment above 85%, demonstrating robust risk-sensitivity and adaptability. This approach offers a practical path toward safer, human-like autonomous driving by coupling rigorous risk assessment with learning-from-experience and reflective reasoning.

Abstract

Recent advancements in autonomous vehicles (AVs) use Large Language Models (LLMs) to perform well in normal driving scenarios. However, ensuring safety in dynamic, high-risk environments and managing safety-critical long-tail events remain significant challenges. To address these issues, we propose SafeDrive, a knowledge- and data-driven risk-sensitive decision-making framework to enhance AV safety and adaptability. The proposed framework introduces a modular system comprising: (1) a Risk Module for quantifying multi-factor coupled risks involving driver, vehicle, and road interactions; (2) a Memory Module for storing and retrieving typical scenarios to improve adaptability; (3) a LLM-powered Reasoning Module for context-aware safety decision-making; and (4) a Reflection Module for refining decisions through iterative learning. By integrating knowledge-driven insights with adaptive learning mechanisms, the framework ensures robust decision-making under uncertain conditions. Extensive evaluations on real-world traffic datasets, including highways (HighD), intersections (InD), and roundabouts (RounD), validate the framework's ability to enhance decision-making safety (achieving a 100% safety rate), replicate human-like driving behaviors (with decision alignment exceeding 85%), and adapt effectively to unpredictable scenarios. SafeDrive establishes a novel paradigm for integrating knowledge- and data-driven methods, highlighting significant potential to improve safety and adaptability of autonomous driving in high-risk traffic scenarios. Project Page: https://mezzi33.github.io/SafeDrive/

Figures (10)

• Figure 1: Overview of SafeDrive: the knowledge- and data-driven risk-sensitive decision-making framework. Input: Hand-labeled high-risk scenario descriptions from real-world datasets. Process: The coupled risk quantification model generates multi-dimensional risk assessments based on real-time trajectory data. The data-driven scenario descriptions, combined with risk prior knowledge and past experiences, are processed by the LLM-based agent using chain-of-thought (CoT) reasoning for adaptive decision-making. Output: An action decision (accelerate, decelerate, change lanes, or idle) generated by the LLM agent, checked through a reflection module for correction, and stored in the memory database for future retrieval, enabling continuous improvement.
• Figure 2: The omnidirectional risk quantification of surrounding vehicles: QPR attributes include the risk cost of each vehicle that has a mutual impact with the ego vehicle, based on the DRF distribution.
• Figure 3: SafeDrive in real-world scenarios: The multi-dimensional risk quantification model captures comprehensive risk information from the surroundings. By combining this risk model with the closed-loop LLM-based driving agent, the system ensures both immediate risk prevention and implements long-term safety measures, ensuring real-time decision-making safety across diverse scenarios. The distributions of QPRs on highway and urban intersections are presented at the top left corner, and corresponding risk-levels of the driving scenes showcase the effectiveness and consistency of our model.
• Figure 4: Example of system prompts and interaction within the SafeDrive system.
• Figure 5: Risk quantification heatmap in diverse driving scenarios: HighD, RounD, InD.