EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction

TL;DR

The paper defines AVEP, a vision-driven version of script-style event prediction that uses structured multimodal event graphs. It introduces EventFormer, a node-graph hierarchical attention Transformer, augmented with a coreference encoding scheme, and tunes a two-stage training pipeline to predict future event triggers and arguments from historical event graphs. A new AVEP dataset (around $35K$ videos and $178K$ event graphs) supports the task, with detailed construction, statistics, and bounding-box annotations. Empirical results show state-of-the-art performance against both video-prediction baselines and large vision-language models, highlighting the importance of modeling event structure and cross-event coreference for open-ended video reasoning.

Abstract

Script event induction, which aims to predict the subsequent event based on the context, is a challenging task in NLP, achieving remarkable success in practical applications. However, human events are mostly recorded and presented in the form of videos rather than scripts, yet there is a lack of related research in the realm of vision. To address this problem, we introduce AVEP (Action-centric Video Event Prediction), a task that distinguishes itself from existing video prediction tasks through its incorporation of more complex logic and richer semantic information. We present a large structured dataset, which consists of about $35K$ annotated videos and more than $178K$ video clips of event, built upon existing video event datasets to support this task. The dataset offers more fine-grained annotations, where the atomic unit is represented as a multimodal event argument node, providing better structured representations of video events. Due to the complexity of event structures, traditional visual models that take patches or frames as input are not well-suited for AVEP. We propose EventFormer, a node-graph hierarchical attention based video event prediction model, which can capture both the relationships between events and their arguments and the coreferencial relationships between arguments. We conducted experiments using several SOTA video prediction models as well as LVLMs on AVEP, demonstrating both the complexity of the task and the value of the dataset. Our approach outperforms all these video prediction models. We will release the dataset and code for replicating the experiments and annotations.

Figures (6)

• Figure 1: The architecture of EventFormer. On the left is the overall model framework, which follows the same architecture as Transformer. On the right is a magnified view of Node-graph Hierarchical Attention, providing detailed explanations of its implementation. We include calculation examples for these components in the Appendix.
• Figure 2: The architecture of Video Event Prediction Model based on EventFormer. The left side of the image shows the overall framework of the model. The frozen symbols indicate that the model's parameters are frozen during training. The right side of the image provides an detailed view of the Coreference Encoding module. The same colored borders represent coreference relationships, while the dashed borders indicate that future events are replaced by the learned mask.
• Figure 3: The annotation pipeline of AVEP. We employ the pre-trained Grounding Dino model to localize the corresponding regions in each frame based on textual annotations. Subsequently, three annotators manually verify and refine the bounding boxes according to the textual descriptions. Finally, the selected regions are cropped to generate a sequence of images, which serve as the visual representations of event graph nodes.
• Figure 4: The data statistics on verbs and nouns in the AVEP dataset. Given the large number of verb categories ($2,284$ in total), we analyzed the distribution by calculating the proportion of the top 30 most frequent verbs versus all others, and also reported the frequency of each verb within the top 30. A similar analysis was conducted for the nouns in the dataset.
• Figure 5: An example illustrating how the attention mechanism is calculated.