Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering

David Romero, Thamar Solorio

TL;DR

Q-ViD presents a simple zero-shot video QA framework that uses question-conditioned frame captions generated by an instruction-tuned vision-language model (InstructBLIP) and a text-based reasoning step (Flan-T5) to answer multiple-choice questions. By transforming video understanding into a text QA problem, it avoids GPT-based pipelines and heavy multi-module architectures yet achieves competitive or superior performance across multiple benchmarks (NExT-QA, STAR, How2QA, TVQA, IntentQA). The key finding is that dedicated question-dependent caption prompts for frame descriptions drive performance gains far more than complex QA prompts, highlighting the effectiveness of targeted visual descriptions for downstream reasoning. The approach is model-agnostic and cost-efficient, with strong zero-shot results and clear ablations that emphasize the importance of task-specific captions over generic descriptions. This has practical implications for deploying accessible, open-model video QA systems without relying on closed GPT-based APIs.

Abstract

We present Q-ViD, a simple approach for video question answering (video QA), that unlike prior methods, which are based on complex architectures, computationally expensive pipelines or use closed models like GPTs, Q-ViD relies on a single instruction-aware open vision-language model (InstructBLIP) to tackle videoQA using frame descriptions. Specifically, we create captioning instruction prompts that rely on the target questions about the videos and leverage InstructBLIP to obtain video frame captions that are useful to the task at hand. Subsequently, we form descriptions of the whole video using the question-dependent frame captions, and feed that information, along with a question-answering prompt, to a large language model (LLM). The LLM is our reasoning module, and performs the final step of multiple-choice QA. Our simple Q-ViD framework achieves competitive or even higher performances than current state of the art models on a diverse range of videoQA benchmarks, including NExT-QA, STAR, How2QA, TVQA and IntentQA.

Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering

TL;DR

Q-ViD presents a simple zero-shot video QA framework that uses question-conditioned frame captions generated by an instruction-tuned vision-language model (InstructBLIP) and a text-based reasoning step (Flan-T5) to answer multiple-choice questions. By transforming video understanding into a text QA problem, it avoids GPT-based pipelines and heavy multi-module architectures yet achieves competitive or superior performance across multiple benchmarks (NExT-QA, STAR, How2QA, TVQA, IntentQA). The key finding is that dedicated question-dependent caption prompts for frame descriptions drive performance gains far more than complex QA prompts, highlighting the effectiveness of targeted visual descriptions for downstream reasoning. The approach is model-agnostic and cost-efficient, with strong zero-shot results and clear ablations that emphasize the importance of task-specific captions over generic descriptions. This has practical implications for deploying accessible, open-model video QA systems without relying on closed GPT-based APIs.

Abstract

We present Q-ViD, a simple approach for video question answering (video QA), that unlike prior methods, which are based on complex architectures, computationally expensive pipelines or use closed models like GPTs, Q-ViD relies on a single instruction-aware open vision-language model (InstructBLIP) to tackle videoQA using frame descriptions. Specifically, we create captioning instruction prompts that rely on the target questions about the videos and leverage InstructBLIP to obtain video frame captions that are useful to the task at hand. Subsequently, we form descriptions of the whole video using the question-dependent frame captions, and feed that information, along with a question-answering prompt, to a large language model (LLM). The LLM is our reasoning module, and performs the final step of multiple-choice QA. Our simple Q-ViD framework achieves competitive or even higher performances than current state of the art models on a diverse range of videoQA benchmarks, including NExT-QA, STAR, How2QA, TVQA and IntentQA.
Paper Structure (18 sections, 4 figures, 4 tables)

This paper contains 18 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Pretraining for Video QA
  4. Image Captions for Video Understanding
  5. Method
  6. Preliminaries: InstructBLIP
  7. Q-ViD: Generating Frame Descriptions for Video QA
  8. Q-ViD: Reasoning Module
  9. Q-ViD: Prompt Design
  10. Experiments
  11. Datasets
  12. Implementation Details
  13. Overall Performance
  14. Ablation Studies
  15. Prompt Analysis
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Overview of Q-ViD. We propose relying on a instructed-tuned multimodal model to generate question-dependent frame captions to perform video QA using text. This simple approach achieves competitive results with more complex architectures or GPT-based methods.
  • Figure 2: Our pipeline for Zero-shot Video QA. Q-ViD prompts InstructBLIP, to obtain video frame descriptions that are tailored to the question needing answer.
  • Figure 3: Variation of captioning templates. We focus on comparing general and question-dependent captioning prompts (Top). For both cases we use the same Base QA instruction template (Bottom).
  • Figure 4: Variation of QA prompt templates. We focus on exploring two more complex and detailed variations for the QA instruction prompt (Bottom). We use the best captioning templates (Top) for Q-ViDXL (General) and Q-ViDXXL (Dependent).