Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Reasoning with Vision-Language Models for Incident Reports from Dashcam Videos

Shingo Yokoi, Kento Sasaki, Yu Yamaguchi

TL;DR

This work tackles incident report generation from dashcam videos under out-of-distribution hazards by proposing a hierarchical framework that decomposes the task into frame-level captioning, incident-frame detection, and incident captioning using vision-language models. It formalizes the 2COOOL challenge with an annotation protocol, defines prerequisite tasks, and uses a combination of automatic metrics and blind human evaluation. The method employs staged processing, gaze augmentation, LLM-based frame selection, and an ensembling + Blind A/B Scoring strategy to improve factuality and readability. Empirically, it ranks 2nd on the open 2COOOL leaderboard and achieves the best CIDEr-D score, demonstrating the potential of VLM-driven hierarchical reasoning for safety-critical traffic event understanding. The work provides a practical pathway toward reliable, interpretable incident narratives in autonomous driving systems.

Abstract

Recent advances in end-to-end (E2E) autonomous driving have been enabled by training on diverse large-scale driving datasets, yet autonomous driving models still struggle in out-of-distribution (OOD) scenarios. The COOOL benchmark targets this gap by encouraging hazard understanding beyond closed taxonomies, and the 2COOOL challenge extends it to generating human-interpretable incident reports. We present a hierarchical reasoning framework for incident report generation from dashcam videos that integrates frame-level captioning, incident frame detection, and fine-grained reasoning within vision-language models (VLMs). We further improve factual accuracy and readability through model ensembling and a Blind A/B Scoring selection protocol. On the official 2COOOL open leaderboard, our method ranks 2nd among 29 teams and achieves the best CIDEr-D score, producing accurate and coherent incident narratives. These results indicate that hierarchical reasoning with VLMs is a promising direction for accident analysis and for broader understanding of safety-critical traffic events. The implementation and code are available at https://github.com/riron1206/kaggle-2COOOL-2nd-Place-Solution.

Hierarchical Reasoning with Vision-Language Models for Incident Reports from Dashcam Videos

TL;DR

This work tackles incident report generation from dashcam videos under out-of-distribution hazards by proposing a hierarchical framework that decomposes the task into frame-level captioning, incident-frame detection, and incident captioning using vision-language models. It formalizes the 2COOOL challenge with an annotation protocol, defines prerequisite tasks, and uses a combination of automatic metrics and blind human evaluation. The method employs staged processing, gaze augmentation, LLM-based frame selection, and an ensembling + Blind A/B Scoring strategy to improve factuality and readability. Empirically, it ranks 2nd on the open 2COOOL leaderboard and achieves the best CIDEr-D score, demonstrating the potential of VLM-driven hierarchical reasoning for safety-critical traffic event understanding. The work provides a practical pathway toward reliable, interpretable incident narratives in autonomous driving systems.

Abstract

Recent advances in end-to-end (E2E) autonomous driving have been enabled by training on diverse large-scale driving datasets, yet autonomous driving models still struggle in out-of-distribution (OOD) scenarios. The COOOL benchmark targets this gap by encouraging hazard understanding beyond closed taxonomies, and the 2COOOL challenge extends it to generating human-interpretable incident reports. We present a hierarchical reasoning framework for incident report generation from dashcam videos that integrates frame-level captioning, incident frame detection, and fine-grained reasoning within vision-language models (VLMs). We further improve factual accuracy and readability through model ensembling and a Blind A/B Scoring selection protocol. On the official 2COOOL open leaderboard, our method ranks 2nd among 29 teams and achieves the best CIDEr-D score, producing accurate and coherent incident narratives. These results indicate that hierarchical reasoning with VLMs is a promising direction for accident analysis and for broader understanding of safety-critical traffic events. The implementation and code are available at https://github.com/riron1206/kaggle-2COOOL-2nd-Place-Solution.
Paper Structure (15 sections, 1 equation, 3 figures, 3 tables)

This paper contains 15 sections, 1 equation, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. 2COOOL Challenge
  3. Annotation Protocol
  4. Tasks
  5. Evaluation Metrics
  6. Method
  7. Hierarchical Reasoning Framework
  8. Stage 1: Frame-level Captioning
  9. Stage 2: Incident Frame Detection
  10. Stage 3: Incident Captioning
  11. Implementation Details
  12. Ensembling
  13. Blind A/B Scoring
  14. Results
  15. Conclusion

Figures (3)

  • Figure 1: Overview of our proposed hierarchical reasoning framework. The pipeline consists of three stages: (i) frame-level captioning, (ii) incident frame detection, and (iii) incident captioning, which together generate coherent incident reports from dashcam videos.
  • Figure 2: Interface of the web application for blind A/B scoring, where two reports are shown in random order and human evaluators select the more accurate one.
  • Figure 3: Qualitative examples of incident reports generated by our method. Most reports are factually accurate and coherent, such as (a) and (b), but occasional errors occur in relative spatial orientation, such as confusing right with left (c).