Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Glance and Focus: Memory Prompting for Multi-Event Video Question Answering

Ziyi Bai, Ruiping Wang, Xilin Chen

TL;DR

The paper tackles Multi-Event VideoQA, where questions require reasoning over long videos with multiple events. It introduces Glance-Focus, a two-stage Transformer-based framework that first generates dynamic event memories (glance) and then uses memory prompts in a focusing stage with multi-level cross-attention to locate relevant moments and answer questions. Memory memories can be generated unsupervised with certainty and diversity constraints or obtained via supervised matching when event labels exist, enabling flexible training. Empirical results across STAR, AGQA v2, EgoTaskQA, and NExT-QA achieve state-of-the-art performance, with strong performance even without explicit event annotations, and code is available to facilitate further research.

Abstract

Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance-Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross-attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.

Glance and Focus: Memory Prompting for Multi-Event Video Question Answering

TL;DR

The paper tackles Multi-Event VideoQA, where questions require reasoning over long videos with multiple events. It introduces Glance-Focus, a two-stage Transformer-based framework that first generates dynamic event memories (glance) and then uses memory prompts in a focusing stage with multi-level cross-attention to locate relevant moments and answer questions. Memory memories can be generated unsupervised with certainty and diversity constraints or obtained via supervised matching when event labels exist, enabling flexible training. Empirical results across STAR, AGQA v2, EgoTaskQA, and NExT-QA achieve state-of-the-art performance, with strong performance even without explicit event annotations, and code is available to facilitate further research.

Abstract

Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance-Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross-attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.
Paper Structure (16 sections, 11 equations, 3 figures, 7 tables)

This paper contains 16 sections, 11 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Video Question Answering
  4. Video-Language Pre-trained Models
  5. Multi-Event Modeling
  6. Approach: The Glance-Focus Model
  7. Overview
  8. Glancing Stage
  9. Focusing Stage
  10. Loss Functions
  11. Evaluation
  12. Datasets
  13. Implementation Details
  14. Comparison with State-of-the-arts
  15. Model Analysis
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Comparison of different VideoQA solutions. The cross-modal common feature space is visualized. (a) Cross-modal correspondence modeling. Weak correspondence between the question with high-level event concepts and low-level lengthy video leads to a wrong answer. (b) Human-like Glance-Focus Strategy. Glancing stage: a set of episode memories are generated. Focusing stage: memory prompts are used as anchors to quickly locate the keyframes for reasoning.
  • Figure 2: The complex multi-event reasoning task is structured into two stages: Glancing and Focusing. At the Glancing stage, a set of unordered event memories are obtained by aggregating relevant input video frames. At the Focusing stage, these event memories are reorganized according to their sequential order. The resulting memory prompts act as bridges, allowing model progressively focus on the related video frames for answer prediction using multi-level cross-attention mechanism.
  • Figure 3: Visualization of the event memory generation and the multi-level attention map.