Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AVD2: Accident Video Diffusion for Accident Video Description

Cheng Li, Keyuan Zhou, Tong Liu, Yu Wang, Mingqiao Zhuang, Huan-ang Gao, Bu Jin, Hao Zhao

TL;DR

AVD2 tackles the challenge of understanding traffic accidents in autonomous driving by generating accident videos aligned with detailed descriptive reasoning and preventive guidance. The approach combines a dataset augmentation pipeline (MM-AU to EMM-AU) with a SwinBERT-based captioning framework enhanced by Self-Critical Sequence Training to produce action/cause descriptions and avoidance strategies. Empirical results show state-of-the-art performance across automated metrics and human evaluations, validating improvements in accident analysis and prevention. The work introduces EMM-AU and demonstrates the practical impact of synthetic, high-quality accident videos for advancing safe autonomous driving systems.

Abstract

Traffic accidents present complex challenges for autonomous driving, often featuring unpredictable scenarios that hinder accurate system interpretation and responses. Nonetheless, prevailing methodologies fall short in elucidating the causes of accidents and proposing preventive measures due to the paucity of training data specific to accident scenarios. In this work, we introduce AVD2 (Accident Video Diffusion for Accident Video Description), a novel framework that enhances accident scene understanding by generating accident videos that aligned with detailed natural language descriptions and reasoning, resulting in the contributed EMM-AU (Enhanced Multi-Modal Accident Video Understanding) dataset. Empirical results reveal that the integration of the EMM-AU dataset establishes state-of-the-art performance across both automated metrics and human evaluations, markedly advancing the domains of accident analysis and prevention. Project resources are available at https://an-answer-tree.github.io

AVD2: Accident Video Diffusion for Accident Video Description

TL;DR

AVD2 tackles the challenge of understanding traffic accidents in autonomous driving by generating accident videos aligned with detailed descriptive reasoning and preventive guidance. The approach combines a dataset augmentation pipeline (MM-AU to EMM-AU) with a SwinBERT-based captioning framework enhanced by Self-Critical Sequence Training to produce action/cause descriptions and avoidance strategies. Empirical results show state-of-the-art performance across automated metrics and human evaluations, validating improvements in accident analysis and prevention. The work introduces EMM-AU and demonstrates the practical impact of synthetic, high-quality accident videos for advancing safe autonomous driving systems.

Abstract

Traffic accidents present complex challenges for autonomous driving, often featuring unpredictable scenarios that hinder accurate system interpretation and responses. Nonetheless, prevailing methodologies fall short in elucidating the causes of accidents and proposing preventive measures due to the paucity of training data specific to accident scenarios. In this work, we introduce AVD2 (Accident Video Diffusion for Accident Video Description), a novel framework that enhances accident scene understanding by generating accident videos that aligned with detailed natural language descriptions and reasoning, resulting in the contributed EMM-AU (Enhanced Multi-Modal Accident Video Understanding) dataset. Empirical results reveal that the integration of the EMM-AU dataset establishes state-of-the-art performance across both automated metrics and human evaluations, markedly advancing the domains of accident analysis and prevention. Project resources are available at https://an-answer-tree.github.io

Paper Structure

This paper contains 20 sections, 2 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Interpretability and Accident Analysis in Autonomous Driving
  4. Video Captioning for Autonomous Driving
  5. Video Generation
  6. Methodology
  7. Augmentation and Enhancement of Video Dataset
  8. The MM-AU Dataset
  9. The introduction of EMM-AU Dataset
  10. Super-Resolution for Video Quality Enhancement
  11. AVD2 System
  12. Overview
  13. Action-aware Driving Caption Transformer
  14. Self Critical Sequence Training
  15. Experiment
  16. ...and 5 more sections

Figures (4)

  • Figure 1: The Framework Architecture of AVD2 system.The frame diagram demonstrates a visual language system for generating descriptions and obstacle avoidance cues from video. SwinBERT processes the video input, converts frames into video tags, and outputs descriptions and obstacle avoidance suggestions via a text generation module. The description includes the driving situation of the vehicle, and the obstacle avoidance section gives safety suggestions. Visual-language Transformer extracts text and image features and optimizes the generation with SCST.
  • Figure 2: The Incident Frames of Video Generated by original Open-Sora Model. The prompts used the original MM-AU dataset annotations.
  • Figure 3: Visualization for the First Video Understanding
  • Figure 4: Visualization for the Second Video Understanding