Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-object event graph representation learning for Video Question Answering

Yanan Wang, Shuichiro Haruta, Donghuo Zeng, Julio Vizcarra, Mori Kurokawa

TL;DR

This work proposes a contrastive language event graph representation learning method called CLanG, aiming to capture event representations associated with multiple objects, that outperforms a strong baseline in handling causal and temporal questions and is strength in reasoning multiple object-based events.

Abstract

Video question answering (VideoQA) is a task to predict the correct answer to questions posed about a given video. The system must comprehend spatial and temporal relationships among objects extracted from videos to perform causal and temporal reasoning. While prior works have focused on modeling individual object movements using transformer-based methods, they falter when capturing complex scenarios involving multiple objects (e.g., "a boy is throwing a ball in a hoop"). We propose a contrastive language event graph representation learning method called CLanG to address this limitation. Aiming to capture event representations associated with multiple objects, our method employs a multi-layer GNN-cluster module for adversarial graph representation learning, enabling contrastive learning between the question text and its relevant multi-object event graph. Our method outperforms a strong baseline, achieving up to 2.2% higher accuracy on two challenging VideoQA datasets, NExT-QA and TGIF-QA-R. In particular, it is 2.8% better than baselines in handling causal and temporal questions, highlighting its strength in reasoning multiple object-based events.

Multi-object event graph representation learning for Video Question Answering

TL;DR

This work proposes a contrastive language event graph representation learning method called CLanG, aiming to capture event representations associated with multiple objects, that outperforms a strong baseline in handling causal and temporal questions and is strength in reasoning multiple object-based events.

Abstract

Video question answering (VideoQA) is a task to predict the correct answer to questions posed about a given video. The system must comprehend spatial and temporal relationships among objects extracted from videos to perform causal and temporal reasoning. While prior works have focused on modeling individual object movements using transformer-based methods, they falter when capturing complex scenarios involving multiple objects (e.g., "a boy is throwing a ball in a hoop"). We propose a contrastive language event graph representation learning method called CLanG to address this limitation. Aiming to capture event representations associated with multiple objects, our method employs a multi-layer GNN-cluster module for adversarial graph representation learning, enabling contrastive learning between the question text and its relevant multi-object event graph. Our method outperforms a strong baseline, achieving up to 2.2% higher accuracy on two challenging VideoQA datasets, NExT-QA and TGIF-QA-R. In particular, it is 2.8% better than baselines in handling causal and temporal questions, highlighting its strength in reasoning multiple object-based events.
Paper Structure (14 sections, 9 equations, 2 figures, 5 tables)

This paper contains 14 sections, 9 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Multi-object event graph Initialization
  5. Multi-layer GNN-cluster module
  6. Adversarial graph representation learning
  7. Language event graph contrastive learning
  8. Video question answering
  9. Experiments
  10. Dataset
  11. Training details
  12. Performance
  13. Ablation study
  14. Conclusion

Figures (2)

  • Figure 1: Overview of CLanG. Our proposed method obtains multi-object hierarchical event graph representations for causal and temporal reasoning in videos by giving a fully connected multi-object event graph.
  • Figure 2: CLanG architecture. We first initialize a dense adjacency matrix with encoded object nodes to build a multi-object event graph, then we apply a multi-layer GNN-cluster module with a discriminator for adversarial graph representation learning, where the multi-layer GNN-cluster module consists of nine GNN-cluster involving the different number of object nodes. Finally, We perform language event graph contrastive learning and video question answering with encoded text and multi-object event graph representations.