Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering

TL;DR

HeurVidQA is introduced, a framework that leverages domain-specific entity-action heuristics to refine pre-trained video-language foundation models and treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model’s focus toward precise cues that enhance reasoning.

Abstract

Video Question Answering (VideoQA) represents a crucial intersection between video understanding and language processing, requiring both discriminative unimodal comprehension and sophisticated cross-modal interaction for accurate inference. Despite advancements in multi-modal pre-trained models and video-language foundation models, these systems often struggle with domain-specific VideoQA due to their generalized pre-training objectives. Addressing this gap necessitates bridging the divide between broad cross-modal knowledge and the specific inference demands of VideoQA tasks. To this end, we introduce HeurVidQA, a framework that leverages domain-specific entity-action heuristics to refine pre-trained video-language foundation models. Our approach treats these models as implicit knowledge engines, employing domain-specific entity-action prompters to direct the model's focus toward precise cues that enhance reasoning. By delivering fine-grained heuristics, we improve the model's ability to identify and interpret key entities and actions, thereby enhancing its reasoning capabilities. Extensive evaluations across multiple VideoQA datasets demonstrate that our method significantly outperforms existing models, underscoring the importance of integrating domain-specific knowledge into video-language models for more accurate and context-aware VideoQA.

Figures (9)

• Figure 1: Comparison of Recent Advanced VideoQA Models with the Proposed HeurVidQA Framework: (1) Recent methods utilize VFMs to generate general, domain-agnostic representations, complemented by task-specific heads for question-answering. (2) Our HeurVidQA framework enhances VFMs by integrating domain-specific entity-action prompts, blending implicit general knowledge with fine-grained domain-specific insights, to improve cross-modal understanding and reasoning in VideoQA tasks.
• Figure 2: Overview of the Proposed HeurVidQA Framework. The framework consists of two primary stages: Entity-Action Heuristic Generation and Heuristic-Enhanced Answer Inference. In the first stage, EAPrompter extracts action heuristics through temporal-aware action detection and entity heuristics via spatial-aware entity detection using spatial-temporal-aware crops and instantiated prompt templates. These heuristics, sourced from a predefined vocabulary, include confidence scores for each detected action and entity. The second stage utilizes a Vision-Language Foundation Model (VFM) to process multimodal features through self-attention and cross-attention modules, resulting in a semantically enriched cross-modal fusion that supports both final answer prediction and anticipation of actions and entities in the video.
• Figure 3: Training Framework of EAPrompter. The Symmetric Contrastive Loss is employed to jointly train a video encoder and a text encoder, facilitating the prediction of correct pairings within batches of {video, text} samples.
• Figure 4: Operational Principle of Action Prompter. The Action Prompter processes instantiated templates alongside sparsely selected video frames, which have been refined through a space-time cropping strategy.
• Figure 5: Visualizations of heuristics generated by EAPrompter. We selected commonly used verbs and nouns as candidates for action and entity prompts, respectively, and labeled the top five with their associated probabilities (in brackets). To obtain heuristics for actions (a) and entities (b), we applied distinct processing strategies: consistent regions across frames for action prompts, and varying regions for entity prompts to mimic real-world scenarios. Labels with a probability below 10% were filtered out, with significant heuristics highlighted in red.