Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question Answering

TL;DR

LGQAVE tackles VideoQA by tightly coupling question semantics with frame relevance and object-level grounding. It introduces question-guided frame sampling, MiniGPT-4 grounded object graphs, and a question-aware dynamic graph transformer to produce discriminative local and global video representations, which are fused via cross-attention before answer generation by a language model. Empirical results across NExT-QA, TGIF-FrameQA, MSRVTT-QA, ActivityNetQA, and related datasets show state-of-the-art performance and robust gains over both non-LLM and LLM baselines, with ablations validating the contribution of each component. The work demonstrates a scalable, efficient approach to multi-modal reasoning in videos and points to future extensions for longer sequences and more complex queries.

Abstract

This paper tackles the intricate challenge of video question-answering (VideoQA). Despite notable progress, current methods fall short of effectively integrating questions with video frames and semantic object-level abstractions to create question-aware video representations. We introduce Local-Global Question Aware Video Embedding (LGQAVE), which incorporates three major innovations to integrate multi-modal knowledge better and emphasize semantic visual concepts relevant to specific questions. LGQAVE moves beyond traditional ad-hoc frame sampling by utilizing a cross-attention mechanism that precisely identifies the most relevant frames concerning the questions. It captures the dynamics of objects within these frames using distinct graphs, grounding them in question semantics with the miniGPT model. These graphs are processed by a question-aware dynamic graph transformer (Q-DGT), which refines the outputs to develop nuanced global and local video representations. An additional cross-attention module integrates these local and global embeddings to generate the final video embeddings, which a language model uses to generate answers. Extensive evaluations across multiple benchmarks demonstrate that LGQAVE significantly outperforms existing models in delivering accurate multi-choice and open-ended answers.

Figures (4)

• Figure 1: Qualitative analysis of LGQAVE. We present the answers produced by various state-of-the-art VideoQA models in response to a specific question paired with a sequence of frames from a given video in the NextQA nextqa dataset. Our findings indicate that the answers generated by our LGQAVE model are notably more direct and precise in their semantic content.
• Figure 2: Schematic of the model diagram for LGQAVE. Given a question and its corresponding video, our process begins with a question-aware frame sampling module that identifies the pertinent frames from the video. Subsequently, a miniGPT4-based visual grounding module constructs object relation graphs from these selected frames. The Q-DGT module then processes these graphs along with masked question embeddings to produce local and global video representations. A cross-attention module further refines the global features by incorporating contextual knowledge from the local features. Finally, a language model-based answer generator utilizes these refined features to predict the answers.
• Figure 3: Performance of LGQAVE with change in $\beta$ and $\gamma$ parameters on various datasets are shown in the first two plots. Performance of LGQAVE with usage of different combinations of $\mathcal{F}_{{local}}$ and $\mathcal{F}_{{global}}.$ is shown at the end.
• Figure 4: Qualitative answers$^{\dagger}$ by LGQAVE model for various ablation configurations on a video from the NextQA dataset.