Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey of Video Datasets for Grounded Event Understanding

Kate Sanders, Benjamin Van Durme

TL;DR

The paper tackles the lack of a unified view on video event understanding by surveying 105 event-centric video datasets and proposing a three-axis framework focused on content, presentation, and structure. It introduces a taxonomy of video event structures and analyzes how datasets align with emerging video event extraction tasks, highlighting the importance of temporal dynamics, hierarchical event modeling, and uncertainty in robust reasoning. The authors argue that combining formal semantic representations with natural language descriptions, along with diverse and multi-format data, is essential for grounding video events in vision-language models. They offer concrete recommendations for dataset construction and task framing to advance robust, human-like event understanding in unconstrained video inputs.

Abstract

While existing video benchmarks largely consider specialized downstream tasks like retrieval or question-answering (QA), contemporary multimodal AI systems must be capable of well-rounded common-sense reasoning akin to human visual understanding. A critical component of human temporal-visual perception is our ability to identify and cognitively model "things happening", or events. Historically, video benchmark tasks have implicitly tested for this ability (e.g., video captioning, in which models describe visual events with natural language), but they do not consider video event understanding as a task in itself. Recent work has begun to explore video analogues to textual event extraction but consists of competing task definitions and datasets limited to highly specific event types. Therefore, while there is a rich domain of event-centric video research spanning the past 10+ years, it is unclear how video event understanding should be framed and what resources we have to study it. In this paper, we survey 105 video datasets that require event understanding capability, consider how they contribute to the study of robust event understanding in video, and assess proposed video event extraction tasks in the context of this body of research. We propose suggestions informed by this survey for dataset curation and task framing, with an emphasis on the uniquely temporal nature of video events and ambiguity in visual content.

A Survey of Video Datasets for Grounded Event Understanding

TL;DR

The paper tackles the lack of a unified view on video event understanding by surveying 105 event-centric video datasets and proposing a three-axis framework focused on content, presentation, and structure. It introduces a taxonomy of video event structures and analyzes how datasets align with emerging video event extraction tasks, highlighting the importance of temporal dynamics, hierarchical event modeling, and uncertainty in robust reasoning. The authors argue that combining formal semantic representations with natural language descriptions, along with diverse and multi-format data, is essential for grounding video events in vision-language models. They offer concrete recommendations for dataset construction and task framing to advance robust, human-like event understanding in unconstrained video inputs.

Abstract

While existing video benchmarks largely consider specialized downstream tasks like retrieval or question-answering (QA), contemporary multimodal AI systems must be capable of well-rounded common-sense reasoning akin to human visual understanding. A critical component of human temporal-visual perception is our ability to identify and cognitively model "things happening", or events. Historically, video benchmark tasks have implicitly tested for this ability (e.g., video captioning, in which models describe visual events with natural language), but they do not consider video event understanding as a task in itself. Recent work has begun to explore video analogues to textual event extraction but consists of competing task definitions and datasets limited to highly specific event types. Therefore, while there is a rich domain of event-centric video research spanning the past 10+ years, it is unclear how video event understanding should be framed and what resources we have to study it. In this paper, we survey 105 video datasets that require event understanding capability, consider how they contribute to the study of robust event understanding in video, and assess proposed video event extraction tasks in the context of this body of research. We propose suggestions informed by this survey for dataset curation and task framing, with an emphasis on the uniquely temporal nature of video events and ambiguity in visual content.
Paper Structure (21 sections, 8 equations, 2 figures, 1 table)

This paper contains 21 sections, 8 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Survey Methodology
  3. Dataset Content
  4. Dataset Presentation
  5. Dataset Structure
  6. Dataset Collection
  7. Video Analysis
  8. Results
  9. Discussion
  10. Structure Analysis
  11. Events and Language
  12. Semantic Structure
  13. Semantic Structure: Discussion
  14. Temporal Structure
  15. Temporal Structure: Discussion
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Illustration of an example video dataset paired with the associated topics we cover in our survey, detailed in Section 2. These topics are designed to help us answer the question of what resources we have for robust video event understanding, targeting the content, presentation, and structure of video events.
  • Figure 2: Illustration of the two axes along which we classify event structures presented in video datasets. Alongside each category name, we include a graphic depicting the general idea of the method, a dataset that falls into this structure category, and an image of the dataset presentation to show how these methods exist in real data (dataset images are taken from the original publication). The top row shows how details are organized, and the bottom row shows how time is incorporated into the structure.