Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries

TL;DR

This work tackles Video QA by addressing the challenge of extracting question-relevant information across temporal video frames. It introduces T-Former, a question-guided temporal querying Transformer, to build a temporal bridge between frame-wise perception and LLM-based reasoning within the PQR framework (Perceive, Query, Reason). PQR uses an instruction-aware visual encoder, dynamic temporal queries initialized from video content, and a frozen LLM to reason over condensed visual-temporal representations, trained only on target datasets. Across multiple benchmarks, PQR with T-Former achieves state-of-the-art performance, demonstrating strong temporal and causal reasoning while maintaining computational efficiency. The approach highlights the value of dynamic, question-guided temporal features as a scalable bridge between pre-trained vision-language models and video-specific reasoning tasks.

Abstract

Video Question Answering (Video QA) is a challenging video understanding task that requires models to comprehend entire videos, identify the most relevant information based on contextual cues from a given question, and reason accurately to provide answers. Recent advancements in Multimodal Large Language Models (MLLMs) have transformed video QA by leveraging their exceptional commonsense reasoning capabilities. This progress is largely driven by the effective alignment between visual data and the language space of MLLMs. However, for video QA, an additional space-time alignment poses a considerable challenge for extracting question-relevant information across frames. In this work, we investigate diverse temporal modeling techniques to integrate with MLLMs, aiming to achieve question-guided temporal modeling that leverages pre-trained visual and textual alignment in MLLMs. We propose T-Former, a novel temporal modeling method that creates a question-guided temporal bridge between frame-wise visual perception and the reasoning capabilities of LLMs. Our evaluation across multiple video QA benchmarks demonstrates that T-Former competes favorably with existing temporal modeling approaches and aligns with recent advancements in video QA.

Figures (5)

• Figure 1: Adapting LLMs' visual reasoning capabilities to video QA requires extracting the most relevant video features based on the input question. Our approach addresses this challenge by extracting question-guided temporal features from dynamic frame sequences, enabling accurate and context-aware reasoning.
• Figure 2: Our model consists of three primary stages: Perceive, Query, and Reason. The Perceive stage employs a ViT encoder and a Q-Former to extract spatial visual features from each frame independently, conditioned on the input question. The Query stage introduces T-Former, our question-guided temporal querying Transformer that captures the most relevant temporal information conditioned on both the question and the visual context. Finally, in the Reason stage, the condensed visual-temporal features are integrated with the question and answer options before being fed into a frozen LLM as a reasoning agent to rationalize and answer the question.
• Figure 3: Overview of T-Former, a Temporal Querying Transformer. It adopts 1) a self-attention layer between temporal queries and question queries and 2) cross-attention layers between query tokens and the full-length sequence of visual tokens.
• Figure 4: For different video samples (on each row), we visualize the attention map between question-guided temporal queries $\mathbf{q}$ and framewise visual tokens $\mathbf{e}^{1:n}_f$ in the last layer of T-Former during epochs 1, 5, and 10.
• Figure 5: Exploring Linguistic Bias. We observe that LLM reasoners can only achieve a modest performance, akin to a "blind guess" when visual inputs are absent.