Bridging Human Oversight and Black-box Driver Assistance: Vision-Language Models for Predictive Alerting in Lane Keeping Assist Systems

TL;DR

Unpredictable LKA failures in current ADAS reduce driver trust and safety due to the opaque nature of black-box systems. The authors propose LKAlert, a Vision-Language Model–based supervisory alerting system that predicts LKA risk 1–3 seconds in advance using dash-cam video, CAN data, and LaneNet-based lane guidance, and it outputs both a binary alert and a natural language explanation. A novel OpenLKA-Alert dataset supports predictive, explainable warnings and underpins a generalizable framework that uses surrogate lane features with LoRA-fine-tuned decoders while keeping the vision backbone frozen. Empirical results show LKAlert achieves 69.80% accuracy and 58.63% F1 on LKA failure prediction with ROUGE-L 71.72 for explanations, operating around 2 Hz, establishing its practicality for real-time, driver-facing supervision of black-box ADAS. The work offers a scalable paradigm for applying Vision-Language Models to human-centered supervision of opaque autonomous systems and provides a valuable resource for further research in interpretable safety-critical AI in vehicular contexts.

Abstract

Lane Keeping Assist systems, while increasingly prevalent, often suffer from unpredictable real-world failures, largely due to their opaque, black-box nature, which limits driver anticipation and trust. To bridge the gap between automated assistance and effective human oversight, we present LKAlert, a novel supervisory alert system that leverages VLM to forecast potential LKA risk 1-3 seconds in advance. LKAlert processes dash-cam video and CAN data, integrating surrogate lane segmentation features from a parallel interpretable model as automated guiding attention. Unlike traditional binary classifiers, LKAlert issues both predictive alert and concise natural language explanation, enhancing driver situational awareness and trust. To support the development and evaluation of such systems, we introduce OpenLKA-Alert, the first benchmark dataset designed for predictive and explainable LKA failure warnings. It contains synchronized multimodal inputs and human-authored justifications across annotated temporal windows. We further contribute a generalizable methodological framework for VLM-based black-box behavior prediction, combining surrogate feature guidance with LoRA. This framework enables VLM to reason over structured visual context without altering its vision backbone, making it broadly applicable to other complex, opaque systems requiring interpretable oversight. Empirical results correctly predicts upcoming LKA failures with 69.8% accuracy and a 58.6\% F1-score. The system also generates high-quality textual explanations for drivers (71.7 ROUGE-L) and operates efficiently at approximately 2 Hz, confirming its suitability for real-time, in-vehicle use. Our findings establish LKAlert as a practical solution for enhancing the safety and usability of current ADAS and offer a scalable paradigm for applying VLMs to human-centered supervision of black-box automation.

Figures (7)

• Figure 1: The video file is segmented into video frames every 0.5 second and preprocessed by Lanenet to obtain the intermediate process of lane line recognition: Binary Segment and Instance Segment, which provide interpretability for edge segmentation and segmentation of different lane lines respectively. The vehicle control information in CAN and the output information of Openpilot are used as text data, which can be provided together with the previous visual data during training.
• Figure 2: Visual Data From OpenLKA-Failure Dataset and OpenLKA-Normal Dataset
• Figure 3: Although the current laneline is clear, there is a sharp curve ahead, and the car will lean to the right, almost depart the lane in 3 seconds, so this data is marked as Alert.
• Figure 4: Semantic word cloud of the LKA Failure Explanation in OpenLKA-Alert
• Figure 5: LKAlert Architecture: Multimodal inputs (Image $\mathcal{I}_{rgb}$, Masks $\mathcal{M}_{bin}, \mathcal{M}_{ins}$, CAN $\mathbf{c}$) are processed by frozen encoders ($f_{ViT}, f_{Text}$). The combined representation $\mathbf{X}$ feeds the LoRA-adapted decoder $f_{Dec}$ to generate the alert $y$ and explanation $e$.