Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Video Question Answering through query-based frame selection

Himanshu Patil, Geo Jolly, Ramana Raja Buddala, Ganesh Ramakrishnan, Rohit Saluja

TL;DR

This work presents a novel query-based selection of frames relevant to the questions based on the submodular mutual Information (SMI) functions, and opine that such query-based frame selection can enhance accuracy in a wide range of tasks that rely on only a subset of video frames.

Abstract

Video Question Answering (VideoQA) models enhance understanding and interaction with audiovisual content, making it more accessible, searchable, and useful for a wide range of fields such as education, surveillance, entertainment, and content creation. Due to heavy compute requirements, most large visual language models (VLMs) for VideoQA rely on a fixed number of frames by uniformly sampling the video. However, this process does not pick important frames or capture the context of the video. We present a novel query-based selection of frames relevant to the questions based on the submodular mutual Information (SMI) functions. By replacing uniform frame sampling with query-based selection, our method ensures that the chosen frames provide complementary and essential visual information for accurate VideoQA. We evaluate our approach on the MVBench dataset, which spans a diverse set of multi-action video tasks. VideoQA accuracy on this dataset was assessed using two VLMs, namely Video-LLaVA and LLaVA-NeXT, both of which originally employed uniform frame sampling. Experiments were conducted using both uniform and query-based sampling strategies. An accuracy improvement of up to \textbf{4\%} was observed when using query-based frame selection over uniform sampling. Qualitative analysis further highlights that query-based selection, using SMI functions, consistently picks frames better aligned with the question. We opine that such query-based frame selection can enhance accuracy in a wide range of tasks that rely on only a subset of video frames.

Improving Video Question Answering through query-based frame selection

TL;DR

This work presents a novel query-based selection of frames relevant to the questions based on the submodular mutual Information (SMI) functions, and opine that such query-based frame selection can enhance accuracy in a wide range of tasks that rely on only a subset of video frames.

Abstract

Video Question Answering (VideoQA) models enhance understanding and interaction with audiovisual content, making it more accessible, searchable, and useful for a wide range of fields such as education, surveillance, entertainment, and content creation. Due to heavy compute requirements, most large visual language models (VLMs) for VideoQA rely on a fixed number of frames by uniformly sampling the video. However, this process does not pick important frames or capture the context of the video. We present a novel query-based selection of frames relevant to the questions based on the submodular mutual Information (SMI) functions. By replacing uniform frame sampling with query-based selection, our method ensures that the chosen frames provide complementary and essential visual information for accurate VideoQA. We evaluate our approach on the MVBench dataset, which spans a diverse set of multi-action video tasks. VideoQA accuracy on this dataset was assessed using two VLMs, namely Video-LLaVA and LLaVA-NeXT, both of which originally employed uniform frame sampling. Experiments were conducted using both uniform and query-based sampling strategies. An accuracy improvement of up to \textbf{4\%} was observed when using query-based frame selection over uniform sampling. Qualitative analysis further highlights that query-based selection, using SMI functions, consistently picks frames better aligned with the question. We opine that such query-based frame selection can enhance accuracy in a wide range of tasks that rely on only a subset of video frames.
Paper Structure (28 sections, 6 equations, 2 figures, 3 tables)

This paper contains 28 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Submodular Optimization in Video Summarization
  4. Vision-Language Models for Video Question Answering
  5. Key frame sampling techniques
  6. Methodology
  7. Submodular functions
  8. Submodular Mutual Information Functions
  9. Implementation Details
  10. Frame and Text embeddings:
  11. Query-based frame selection using SMI functions:
  12. VideoQA using VLMs: Video-LLaVA and LLaVA-NeXT
  13. Training Setup
  14. Video-LLaVA implementation:
  15. LLaVA-NeXT Implementation:
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Figure illustrates how the frames are selected employing Query based selection
  • Figure 2: Figure illustrates uniform sampling and subset selection passed to VideoLLaVAlin-videollava