GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval

TL;DR

The paper tackles the challenge of fine-grained video understanding by focusing on generic event boundaries—moments where dominant subjects undergo status changes. It introduces Kinetic-GEB+, a large-scale boundary-captioning dataset (176,681 boundaries in 12,434 videos) derived from Kinetic-400, with a three-part downstream task suite: Boundary Captioning, Boundary Grounding, and Boundary Caption-Video Retrieval, augmented by a Temporal-based Pairwise Difference (TPD) modeling approach to capture fine-grained visual differences. Across extensive experiments, ActBERT-revised excels at captioning while FROZEN-revised leads in grounding and retrieval, and results consistently highlight the importance of combining instant- and event-granularity features and robust temporal localization. The work also provides detailed annotation protocols, quality-control measures, and analyses of boundary distributions and caption diversity, making Kinetic-GEB+ a valuable resource for advancing open-world, status-change–driven video understanding and downstream intelligent systems.

Abstract

Cognitive science has shown that humans perceive videos in terms of events separated by the state changes of dominant subjects. State changes trigger new events and are one of the most useful among the large amount of redundant information perceived. However, previous research focuses on the overall understanding of segments without evaluating the fine-grained status changes inside. In this paper, we introduce a new dataset called Kinetic-GEB+. The dataset consists of over 170k boundaries associated with captions describing status changes in the generic events in 12K videos. Upon this new dataset, we propose three tasks supporting the development of a more fine-grained, robust, and human-like understanding of videos through status changes. We evaluate many representative baselines in our dataset, where we also design a new TPD (Temporal-based Pairwise Difference) Modeling method for visual difference and achieve significant performance improvements. Besides, the results show there are still formidable challenges for current methods in the utilization of different granularities, representation of visual difference, and the accurate localization of status changes. Further analysis shows that our dataset can drive developing more powerful methods to understand status changes and thus improve video level comprehension. The dataset including both videos and boundaries is available at https://yuxuan-w.github.io/GEB-plus/

Figures (15)

• Figure 1: An example of generic event boundaries with captions in Kinetic-GEB+, as well as three downstream tasks designed upon the boundaries.
• Figure 2: Three samples from Kinetic-GEB+. Each boundary consists of a temporal position and an associated caption, with the boundary type noted at the bottom
• Figure 3: Three candidate formats of Boundary Captions and their evaluation results, respectively One-Sentence format, Two-Item format and Our Finalized format
• Figure 4: Top. Distribution of boundary types in Train/Val/Test split. Bottom Left. Annotation numbers versus the numbers of boundary in the annotation. Bottom Right. Boundary numbers versus the duration of the interval before the boundary
• Figure 5: (a) The parts of speech distribution of the Status Before and Status After compared with that of the Subject part. The two status parts contains more verbs and focus more on motions. (b) The 20 most frequent nouns in Subject. (c) The 20 most frequent verbs in Status Before and Status After