Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Top-down Activity Representation Learning for Video Question Answering

Yanan Wang, Shuichiro Haruta, Donghuo Zeng, Julio Vizcarra, Mori Kurokawa

TL;DR

This paper converts long-term video sequences into a spatial image domain and finetune the multimodal model LLaVA for the VideoQA task to leverage the spatial visual context representation capability of the CLIP model for obtaining non-continuous visual representations in terms of contextual events in videos.

Abstract

Capturing complex hierarchical human activities, from atomic actions (e.g., picking up one present, moving to the sofa, unwrapping the present) to contextual events (e.g., celebrating Christmas) is crucial for achieving high-performance video question answering (VideoQA). Recent works have expanded multimodal models (e.g., CLIP, LLaVA) to process continuous video sequences, enhancing the model's temporal reasoning capabilities. However, these approaches often fail to capture contextual events that can be decomposed into multiple atomic actions non-continuously distributed over relatively long-term sequences. In this paper, to leverage the spatial visual context representation capability of the CLIP model for obtaining non-continuous visual representations in terms of contextual events in videos, we convert long-term video sequences into a spatial image domain and finetune the multimodal model LLaVA for the VideoQA task. Our approach achieves competitive performance on the STAR task, in particular, with a 78.4% accuracy score, exceeding the current state-of-the-art score by 2.8 points on the NExTQA task.

Top-down Activity Representation Learning for Video Question Answering

TL;DR

This paper converts long-term video sequences into a spatial image domain and finetune the multimodal model LLaVA for the VideoQA task to leverage the spatial visual context representation capability of the CLIP model for obtaining non-continuous visual representations in terms of contextual events in videos.

Abstract

Capturing complex hierarchical human activities, from atomic actions (e.g., picking up one present, moving to the sofa, unwrapping the present) to contextual events (e.g., celebrating Christmas) is crucial for achieving high-performance video question answering (VideoQA). Recent works have expanded multimodal models (e.g., CLIP, LLaVA) to process continuous video sequences, enhancing the model's temporal reasoning capabilities. However, these approaches often fail to capture contextual events that can be decomposed into multiple atomic actions non-continuously distributed over relatively long-term sequences. In this paper, to leverage the spatial visual context representation capability of the CLIP model for obtaining non-continuous visual representations in terms of contextual events in videos, we convert long-term video sequences into a spatial image domain and finetune the multimodal model LLaVA for the VideoQA task. Our approach achieves competitive performance on the STAR task, in particular, with a 78.4% accuracy score, exceeding the current state-of-the-art score by 2.8 points on the NExTQA task.
Paper Structure (13 sections, 3 figures, 3 tables)

This paper contains 13 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Top-down video processing
  5. Training network
  6. Experiment
  7. Dataset
  8. Baselines
  9. Results
  10. Case study
  11. Ablation study
  12. Limitation
  13. Conclusion

Figures (3)

  • Figure 1: Comparing different video processing approaches. The top-down approach (ours) can leverage the strong spatial visual context representation capability to obtain effective representations of both contextual events and atomic actions.
  • Figure 2: Overview of the training network. We add a simple top-down video processor in the LLaVA's architecture to enable the finetuning stage for achieving video question reasoning. We froze the visual encoder during the finetuning process and updated the parameters in the projection layer and LLMs.
  • Figure 3: Three cases selected from NExTQA validation set demonstrate the effect of our proposed video processing approach on video scene understanding. Here, we synthesize $4\times 4$ grid images and use them as the input. The explanation comes from the zero-shot model.