VLM-C4L: Continual Core Dataset Learning with Corner Case Optimization via Vision-Language Models for Autonomous Driving

TL;DR

VLM-C4L tackles the problem of robust corner-case handling in autonomous driving by integrating Vision-Language Models with a continual learning framework that replays a core dataset while progressively incorporating new corner-case information. The method uses VLM-guided extraction to curate balanced corner-case samples, a mean-partitioning scheme, and an uncertainty-based core data update to prevent forgetting. Empirical results on Waymo and CODA show marked improvements in corner-case metrics (AP/AR) under conditions like light pollution and fog, with controlled trade-offs to avoid degradation on regular scenarios. This approach advances practical long-term adaptability for autonomous driving systems in diverse, evolving environments.

Abstract

With the widespread adoption and deployment of autonomous driving, handling complex environments has become an unavoidable challenge. Due to the scarcity and diversity of extreme scenario datasets, current autonomous driving models struggle to effectively manage corner cases. This limitation poses a significant safety risk, according to the National Highway Traffic Safety Administration (NHTSA), autonomous vehicle systems have been involved in hundreds of reported crashes annually in the United States, occurred in corner cases like sun glare and fog, which caused a few fatal accident. Furthermore, in order to consistently maintain a robust and reliable autonomous driving system, it is essential for models not only to perform well on routine scenarios but also to adapt to newly emerging scenarios, especially those corner cases that deviate from the norm. This requires a learning mechanism that incrementally integrates new knowledge without degrading previously acquired capabilities. However, to the best of our knowledge, no existing continual learning methods have been proposed to ensure consistent and scalable corner case learning in autonomous driving. To address these limitations, we propose VLM-C4L, a continual learning framework that introduces Vision-Language Models (VLMs) to dynamically optimize and enhance corner case datasets, and VLM-C4L combines VLM-guided high-quality data extraction with a core data replay strategy, enabling the model to incrementally learn from diverse corner cases while preserving performance on previously routine scenarios, thus ensuring long-term stability and adaptability in real-world autonomous driving. We evaluate VLM-C4L on large-scale real-world autonomous driving datasets, including Waymo and the corner case dataset CODA.

Figures (3)

• Figure 1: Continual Update of the AD Model under the VLM-C4L Framework
• Figure 2: Continual Update of Corner Case Dataset under the VLM-C4L Framework
• Figure 3: Representative and Discriminative Samples in Feature Space