LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios

TL;DR

LiloDriver addresses the challenge of long-tail autonomous driving by fusing perception-based scene understanding with a memory-augmented planner generation module and LLM-guided reasoning to enable lifelong learning without retraining. The four-stage architecture continuously augments planning strategies through memory clustering (incremental DBSCAN) and scenario-specific parameter grids, guided by an LLM that reasons over current context and retrieved experiences. Evaluated on nuPlan, LiloDriver outperforms static rule-based and learning-based planners, with notable gains in rare, safety-critical scenarios and evidence of continual improvement as memory accumulates. This approach offers scalable, human-like adaptation for real-world autonomous driving, reducing retraining needs while enhancing robustness and interpretability.

Abstract

Recent advances in autonomous driving research towards motion planners that are robust, safe, and adaptive. However, existing rule-based and data-driven planners lack adaptability to long-tail scenarios, while knowledge-driven methods offer strong reasoning but face challenges in representation, control, and real-world evaluation. To address these challenges, we present LiloDriver, a lifelong learning framework for closed-loop motion planning in long-tail autonomous driving scenarios. By integrating large language models (LLMs) with a memory-augmented planner generation system, LiloDriver continuously adapts to new scenarios without retraining. It features a four-stage architecture including perception, scene encoding, memory-based strategy refinement, and LLM-guided reasoning. Evaluated on the nuPlan benchmark, LiloDriver achieves superior performance in both common and rare driving scenarios, outperforming static rule-based and learning-based planners. Our results highlight the effectiveness of combining structured memory and LLM reasoning to enable scalable, human-like motion planning in real-world autonomous driving. Our code is available at https://github.com/Hyan-Yao/LiloDriver.

Figures (3)

• Figure 1: Comparison of four planning paradigms for autonomous driving across key criteria, including generalization to long-tail scenarios, scalability, robustness, interpretability, and real-world deployability. LiLoDriver demonstrates comprehensive advantages over rule-based, data-driven, and knowledge-driven approaches.
• Figure 2: The overall architecture of LiloDriver comprises four core modules: (1) Environment and Perception, which integrates vectorized maps and agent histories to construct scene context; (2) Scene Encoder, which converts multi-modal perception inputs into latent embeddings for scene representation; (3) Memory and Planner Generation, which organizes clustered scene embeddings and associated few-shot planning experiences for planner adaptation; (4) Reasoning and Execution, which leverages an LLM to select appropriate behavior planners based on the current scenario.
• Figure 3: The demonstration of LiLoDriver in real-world long-tail scenarios. The first row illustrates a left-turning behavior where the vehicle smoothly adjusts its trajectory over time. The second row shows a pedestrian-rich environment where LiLoDriver exhibits cautious and adaptive planning by slowing down, yielding, and resuming motion. This highlights the system’s ability to handle complex, dynamic traffic conditions with human-like decision making.