Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis

Kaidi Liang, Ke Li, Xianbiao Hu, Ruwen Qin

TL;DR

CrashChat addresses multitask traffic crash video analysis by integrating six tasks within a unified multimodal LLM built on VideoLLaMA3. It introduces a task decoupling strategy with two task groups and LoRA-based instruction tuning to inject domain knowledge, achieving state-of-the-art results on crash recognition, localization, and description/reasoning. The dataset combines MM-AU with Nexar and D2City to enable comprehensive evaluation, and experiments show consistent gains over vision-based baselines and general MLLMs, including improvements in BLEU/ROUGE and localization accuracy. The framework offers an end-to-end tool for practical crash video analysis with strong temporal grounding and reduced hallucinations.

Abstract

Automating crash video analysis is essential to leverage the growing availability of driving video data for traffic safety research and accountability attribution in autonomous driving. Crash video analysis is a challenging multitask problem due to the complex spatiotemporal dynamics of crash events in video data and the diverse analytical requirements involved. It requires capabilities spanning crash recognition, temporal grounding, and high-level video understanding. Existing models, however, cannot perform all these tasks within a unified framework, and effective training strategies for such models remain underexplored. To fill these gaps, this paper proposes CrashChat, a multimodal large language model (MLLM) for multitask traffic crash analysis, built upon VideoLLaMA3. CrashChat acquires domain-specific knowledge through instruction fine-tuning and employs a novel multitask learning strategy based on task decoupling and grouping, which maximizes the benefit of joint learning within and across task groups while mitigating negative transfer. Numerical experiments on consolidated public datasets demonstrate that CrashChat consistently outperforms existing MLLMs across model scales and traditional vision-based methods, achieving state-of-the-art performance. It reaches near-perfect accuracy in crash recognition, a 176\% improvement in crash localization, and a 40\% improvement in the more challenging pre-crash localization. Compared to general MLLMs, it substantially enhances textual accuracy and content coverage in crash description and reasoning tasks, with 0.18-0.41 increases in BLEU scores and 0.18-0.42 increases in ROUGE scores. Beyond its strong performance, CrashChat is a convenient, end-to-end analytical tool ready for practical implementation. The dataset and implementation code for CrashChat are available at https://github.com/Liangkd/CrashChat.

CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis

TL;DR

CrashChat addresses multitask traffic crash video analysis by integrating six tasks within a unified multimodal LLM built on VideoLLaMA3. It introduces a task decoupling strategy with two task groups and LoRA-based instruction tuning to inject domain knowledge, achieving state-of-the-art results on crash recognition, localization, and description/reasoning. The dataset combines MM-AU with Nexar and D2City to enable comprehensive evaluation, and experiments show consistent gains over vision-based baselines and general MLLMs, including improvements in BLEU/ROUGE and localization accuracy. The framework offers an end-to-end tool for practical crash video analysis with strong temporal grounding and reduced hallucinations.

Abstract

Automating crash video analysis is essential to leverage the growing availability of driving video data for traffic safety research and accountability attribution in autonomous driving. Crash video analysis is a challenging multitask problem due to the complex spatiotemporal dynamics of crash events in video data and the diverse analytical requirements involved. It requires capabilities spanning crash recognition, temporal grounding, and high-level video understanding. Existing models, however, cannot perform all these tasks within a unified framework, and effective training strategies for such models remain underexplored. To fill these gaps, this paper proposes CrashChat, a multimodal large language model (MLLM) for multitask traffic crash analysis, built upon VideoLLaMA3. CrashChat acquires domain-specific knowledge through instruction fine-tuning and employs a novel multitask learning strategy based on task decoupling and grouping, which maximizes the benefit of joint learning within and across task groups while mitigating negative transfer. Numerical experiments on consolidated public datasets demonstrate that CrashChat consistently outperforms existing MLLMs across model scales and traditional vision-based methods, achieving state-of-the-art performance. It reaches near-perfect accuracy in crash recognition, a 176\% improvement in crash localization, and a 40\% improvement in the more challenging pre-crash localization. Compared to general MLLMs, it substantially enhances textual accuracy and content coverage in crash description and reasoning tasks, with 0.18-0.41 increases in BLEU scores and 0.18-0.42 increases in ROUGE scores. Beyond its strong performance, CrashChat is a convenient, end-to-end analytical tool ready for practical implementation. The dataset and implementation code for CrashChat are available at https://github.com/Liangkd/CrashChat.

Paper Structure

This paper contains 18 sections, 6 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Traffic Crash Datasets
  4. Vision-based Approaches
  5. MLLM-based Approaches
  6. Methodology
  7. Task Decoupling and Grouping
  8. Model Architecture
  9. Training Multitask Models
  10. Implementation Details
  11. Dataset Construction
  12. Performance Metrics
  13. Results and Discussions
  14. Selection of the Baseline MLLM
  15. Multitask Training Strategies
  16. ...and 3 more sections

Figures (4)

  • Figure 1: CrashChat - a multitask multimodal large language model performing six core tasks in support of crash video analysis in a unified way
  • Figure 2: Overview of the CrashChat's model architecture. Video and text inputs are encoded into tokens and jointly processed by the LLM to generate task outputs, with temporal grounding applied only to detected positive samples.
  • Figure 3: Comprehensive performance comparison of CrashChat, traditional vision-based models, and MLLMs across six core crash video analysis tasks
  • Figure 4: Qualitative comparison of linguistic-centric and perception-centric tasks. Red highlights key semantic inconsistencies, and Green indicates strong alignment with the ground truth.