PADriver: Towards Personalized Autonomous Driving

TL;DR

PADriver addresses the need for truly personalized autonomous driving by integrating multi-modal large language models with a closed-loop decision framework. It introduces a danger level mechanism to quantify risk for each potential action and uses an ego-state history queue and personalized prompts to drive mode-specific behavior (slow, normal, fast). A two-stage training pipeline (pretraining on large rule-based data and SFT on human-derived data) and a new PAD-Highway benchmark built on Highway-Env enable comprehensive evaluation across safety, efficiency, and comfort. Results show that PADriver achieves state-of-the-art performance in slow mode and provides flexible, mode-dependent driving behavior, with the benchmark enabling fair comparison of closed-loop PAD methods and driving modes. This work has practical significance for deploying user-tailored driving styles in autonomous systems and establishes a robust platform for future research and evaluation.

Abstract

In this paper, we propose PADriver, a novel closed-loop framework for personalized autonomous driving (PAD). Built upon Multi-modal Large Language Model (MLLM), PADriver takes streaming frames and personalized textual prompts as inputs. It autoaggressively performs scene understanding, danger level estimation and action decision. The predicted danger level reflects the risk of the potential action and provides an explicit reference for the final action, which corresponds to the preset personalized prompt. Moreover, we construct a closed-loop benchmark named PAD-Highway based on Highway-Env simulator to comprehensively evaluate the decision performance under traffic rules. The dataset contains 250 hours videos with high-quality annotation to facilitate the development of PAD behavior analysis. Experimental results on the constructed benchmark show that PADriver outperforms state-of-the-art approaches on different evaluation metrics, and enables various driving modes.

Figures (4)

• Figure 1: The PADriver framework takes textual prompts(system prompts, personalized prompts and ego state queue) and the BEV image as input, tokenizing them into textual and vision tokens. Based on the multi-modal tokens, LLM then autoregressively generates scene descriptions, assesses the danger level for each potential action, and makes the final action decision. The final action is implicitly affected by both the danger level estimation and the scene description.
• Figure 2: The textual prompts and step-by-step reasoning output are generated by the MLLM in Frame $T$. The textual input includes the system prompts (traffic rules and basic environmental descriptions), personalized prompts, and ego state prompts. The system outputs the scene description, danger level estimation, and the final action output. The model generates the final action <right> in fast mode in frame $t$. The response from the environment occurs in frame $t+1$.
• Figure 3: The distribution of the training dataset, which includes the rule-based set and human-based set, is presented with five actions. From left to right, the first visualization shows the distribution of <KEEP> and other actions using a clustered bar chart. Then, the distribution of non-<KEEP> actions in each set is illustrated in the middle doughnut chart. Lastly, the non-<KEEP> actions of the human-based set are analyzed based on the percentage of three different scores in the right stacked bar chart.
• Figure 4: The illustration of the full input and output of the PADriver.