Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Traffic Safety with Parallel Dense Video Captioning for End-to-End Event Analysis

Maged Shoman, Dongdong Wang, Armstrong Aboah, Mohamed Abdel-Aty

TL;DR

Dense, temporally-grounded description of traffic scenes with pedestrian-vehicle interactions is addressed by an end-to-end PDVC framework using CLIP visual features. The approach integrates domain-specific language modeling and knowledge transfer from BDD-5K to WTS, along with video trimming and an event-counter to produce coherent narratives. Key contributions include a four-component methodology, domain adaptation, and ablation insights that yield competitive performance (6th place) on AI City Challenge Track 2, with open-source code available. This work demonstrates practical, multimodal understanding for traffic safety analysis and provides a transferable baseline for dense video captioning in complex urban scenarios.

Abstract

This paper introduces our solution for Track 2 in AI City Challenge 2024. The task aims to solve traffic safety description and analysis with the dataset of Woven Traffic Safety (WTS), a real-world Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding. Our solution mainly focuses on the following points: 1) To solve dense video captioning, we leverage the framework of dense video captioning with parallel decoding (PDVC) to model visual-language sequences and generate dense caption by chapters for video. 2) Our work leverages CLIP to extract visual features to more efficiently perform cross-modality training between visual and textual representations. 3) We conduct domain-specific model adaptation to mitigate domain shift problem that poses recognition challenge in video understanding. 4) Moreover, we leverage BDD-5K captioned videos to conduct knowledge transfer for better understanding WTS videos and more accurate captioning. Our solution has yielded on the test set, achieving 6th place in the competition. The open source code will be available at https://github.com/UCF-SST-Lab/AICity2024CVPRW

Enhancing Traffic Safety with Parallel Dense Video Captioning for End-to-End Event Analysis

TL;DR

Dense, temporally-grounded description of traffic scenes with pedestrian-vehicle interactions is addressed by an end-to-end PDVC framework using CLIP visual features. The approach integrates domain-specific language modeling and knowledge transfer from BDD-5K to WTS, along with video trimming and an event-counter to produce coherent narratives. Key contributions include a four-component methodology, domain adaptation, and ablation insights that yield competitive performance (6th place) on AI City Challenge Track 2, with open-source code available. This work demonstrates practical, multimodal understanding for traffic safety analysis and provides a transferable baseline for dense video captioning in complex urban scenarios.

Abstract

This paper introduces our solution for Track 2 in AI City Challenge 2024. The task aims to solve traffic safety description and analysis with the dataset of Woven Traffic Safety (WTS), a real-world Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding. Our solution mainly focuses on the following points: 1) To solve dense video captioning, we leverage the framework of dense video captioning with parallel decoding (PDVC) to model visual-language sequences and generate dense caption by chapters for video. 2) Our work leverages CLIP to extract visual features to more efficiently perform cross-modality training between visual and textual representations. 3) We conduct domain-specific model adaptation to mitigate domain shift problem that poses recognition challenge in video understanding. 4) Moreover, we leverage BDD-5K captioned videos to conduct knowledge transfer for better understanding WTS videos and more accurate captioning. Our solution has yielded on the test set, achieving 6th place in the competition. The open source code will be available at https://github.com/UCF-SST-Lab/AICity2024CVPRW
Paper Structure (19 sections, 3 equations, 2 figures, 4 tables)

This paper contains 19 sections, 3 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Dense Video Captioning
  4. Domain-specific Learning
  5. Traffic Safety Description and Analysis
  6. Methodology
  7. Overview
  8. Pre-processing
  9. Feature extraction
  10. PDVC-equipped dense captioning model
  11. Post-processing
  12. Experiments
  13. Dataset
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Overview of our proposed solution. Domain modeling is performed with domain-specific tokenizations for WTS-event, WTS-Normal, and BDD-5K. Caption vehicle (Veh) and caption pedestrian (Ped) models are separately trained with each domain-specific model. To facilitate the caption generation consistency over time, video synchronization is conducted by video trimming. Knowledge transfer from the BDD-5K model to WTS data modeling is conducted to enhance video understanding and caption generation. Subsequently, the models are utilized to infer captions for videos, followed by post-processing that enhances text fluency.
  • Figure 2: A video captioned with the proposed solution. The video is divided into five segments based on event and time, and the best proposal captions are matched to each segment's time frame. Due to space limit, only first two sentences are shown. The time is segmented by seconds.