When, Where, and What? A Novel Benchmark for Accident Anticipation and Localization with Large Language Models

TL;DR

This work tackles accident anticipation with localization in dashcam videos by extending the prediction task to what, when, and where. It introduces a three-stage multimodal framework that fuses visual and object information through a chain-based dynamic diffuse attention mechanism, producing cross-modal inputs for large language models. A novel Stage-3 LLM pipeline (LLaVa-NEXT) delivers verbal accident alerts, enhancing human–AI interaction. Empirical results on DAD, CCD, and A3D demonstrate state-of-the-art AP and mTTA, with strong accident localization (AOLA) and rapid convergence, underscoring practical safety benefits for autonomous driving systems.

Abstract

As autonomous driving systems increasingly become part of daily transportation, the ability to accurately anticipate and mitigate potential traffic accidents is paramount. Traditional accident anticipation models primarily utilizing dashcam videos are adept at predicting when an accident may occur but fall short in localizing the incident and identifying involved entities. Addressing this gap, this study introduces a novel framework that integrates Large Language Models (LLMs) to enhance predictive capabilities across multiple dimensions--what, when, and where accidents might occur. We develop an innovative chain-based attention mechanism that dynamically adjusts to prioritize high-risk elements within complex driving scenes. This mechanism is complemented by a three-stage model that processes outputs from smaller models into detailed multimodal inputs for LLMs, thus enabling a more nuanced understanding of traffic dynamics. Empirical validation on the DAD, CCD, and A3D datasets demonstrates superior performance in Average Precision (AP) and Mean Time-To-Accident (mTTA), establishing new benchmarks for accident prediction technology. Our approach not only advances the technological framework for autonomous driving safety but also enhances human-AI interaction, making predictive insights generated by autonomous systems more intuitive and actionable.

Figures (6)

• Figure 1: Illustration of accident detection, localization, and verbal warning generation performed by our model to enhance safe driving and human-AI interaction. Detected and accident-involved agents are marked as yellow and red bounding boxes, respectively.
• Figure 2: The overall network architecture of our proposed model. It is a cross-modal model including three stages: Feature Extraction and Fusion, Accident Anticipation and Location, and Verbal Accident Alerts.
• Figure 3: Structure of the Dual Vision Attention.
• Figure 4: Structure of the Dynamic Object Attention.
• Figure 5: Comparative analysis of trends in AP and mTTA metrics during training between DSTA (a) and our proposed model (b). The metrics are recorded at every half-epoch.