Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Encoding and Controlling Global Semantics for Long-form Video Question Answering

Thong Thanh Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong-Duy T Nguyen, See-Kiong Ng, Anh Tuan Luu

TL;DR

This work tackles the difficulty of long-form videoQA by introducing a Gated State Space Layer (SSL) within a multi-modal Transformer (GSMT) to inject global video semantics into visual representations. A Cross-modal Compositional Congruence (C^3) objective aligns global visual semantics with the question, enhancing reasoning over extended video sequences. The authors also curate two large-scale long-form benchmarks, Ego-QA and MAD-QA, to rigorously evaluate long-range QA capabilities. Comprehensive experiments, ablations, and qualitative analyses demonstrate substantial gains over strong baselines, highlighting the significance of global semantic integration for real-world, hour-scale videos.

Abstract

Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence (C^3) objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets. The code, model, and data have been made available at https://nguyentthong.github.io/Long_form_VideoQA.

Encoding and Controlling Global Semantics for Long-form Video Question Answering

TL;DR

This work tackles the difficulty of long-form videoQA by introducing a Gated State Space Layer (SSL) within a multi-modal Transformer (GSMT) to inject global video semantics into visual representations. A Cross-modal Compositional Congruence (C^3) objective aligns global visual semantics with the question, enhancing reasoning over extended video sequences. The authors also curate two large-scale long-form benchmarks, Ego-QA and MAD-QA, to rigorously evaluate long-range QA capabilities. Comprehensive experiments, ablations, and qualitative analyses demonstrate substantial gains over strong baselines, highlighting the significance of global semantic integration for real-world, hour-scale videos.

Abstract

Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence (C^3) objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets. The code, model, and data have been made available at https://nguyentthong.github.io/Long_form_VideoQA.
Paper Structure (27 sections, 20 equations, 12 figures, 11 tables)

This paper contains 27 sections, 20 equations, 12 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Gated State Space Multi-Modal Transformer (GSMT)
  5. Cross-modal Compositional Congruence (C$^3$) Objective
  6. Overall Objective
  7. Ego-QA and MAD-QA Benchmarks
  8. Ego-QA
  9. MAD-QA
  10. Experiments
  11. Standard Benchmarks
  12. Implementation Details
  13. Baselines
  14. Quantitative Results
  15. Ablation Study
  16. ...and 12 more sections

Figures (12)

  • Figure 1: Long-form videoQA examples, with videos taken from MAD soldan2022mad and Ego4D grauman2022ego4d datasets, respectively. Question in video 1 requires the model to reason about the relation chain of replacing ingot of palladium to activate the rt unit that powers the armored suit and protects person’s health. Question in video 2 necessitates an understanding of the overall theme in video 2.
  • Figure 2: Illustration of the GSMT architecture empowered by gated SSL and C$^3$ training objective.
  • Figure 3: Qualitative results on the constructed MAD-QA and Ego-QA datasets.
  • Figure 4: GPU memory cost with respect to visual sequence length $L$ of attention, convolution, and gated SSL mechanism.
  • Figure 5: Distribution of video input duration of long-form videoQA datasets.
  • ...and 7 more figures