Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmark

Shaden Shaar, Bradon Thymes, Sirawut Chaixanien, Claire Cardie, Bharath Hariharan

TL;DR

MovieRecapsQA tackles open-ended multimodal VideoQA for long-form content by building a benchmark from recap videos aligned with full movies and introducing a reference-free, fact-grounded evaluation. The dataset comprises about 8,231 QA pairs across 60 films with explicit modality labels and reasoning categories, enabling evaluation of how models ground answers in both video and dialogue. Seven state-of-the-art multimodal LLMs and human annotators are benchmarked, revealing that semantic metrics liberalize model rankings and models rely heavily on dialogue, while visual grounding remains weak. The results highlight a need for architectures and training objectives that explicitly ground language in visual evidence and for evaluation frameworks that rely on atomic facts rather than single gold references.

Abstract

Understanding real-world videos such as movies requires integrating visual and dialogue cues to answer complex questions. Yet existing VideoQA benchmarks struggle to capture this multimodal reasoning and are largely not open-ended, given the difficulty of evaluating free-form answers. In this paper, we introduce a novel open-ended multi-modal VideoQA benchmark, MovieRecapsQA created using movie recap videos--a distinctive type of YouTube content that summarizes a film by presenting its key events through synchronized visual (recap video) and textual (recap summary) modalities. Using the recap summary, we generate $\approx 8.2$ K question-answer (QA) pairs (aligned with movie-subtitles) and provide the necessary "facts" needed to verify an answer in a reference-free manner. To our knowledge, this is the first open-ended VideoQA benchmark that supplies explicit textual context of the input (video and/or text); which we use for evaluation. Our benchmark provides videos of multiple lengths (i.e., recap-segments, movie-segments) and categorizations of questions (by modality and type) to enable fine-grained analysis. We evaluate the performance of seven state-of-the-art MLLMs using our benchmark and observe that: 1) visual-only questions remain the most challenging; 2) models default to textual inputs whenever available; 3) extracting factually accurate information from video content is still difficult for all models; and 4) proprietary and open-source models perform comparably on video-dependent questions.

MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmark

TL;DR

MovieRecapsQA tackles open-ended multimodal VideoQA for long-form content by building a benchmark from recap videos aligned with full movies and introducing a reference-free, fact-grounded evaluation. The dataset comprises about 8,231 QA pairs across 60 films with explicit modality labels and reasoning categories, enabling evaluation of how models ground answers in both video and dialogue. Seven state-of-the-art multimodal LLMs and human annotators are benchmarked, revealing that semantic metrics liberalize model rankings and models rely heavily on dialogue, while visual grounding remains weak. The results highlight a need for architectures and training objectives that explicitly ground language in visual evidence and for evaluation frameworks that rely on atomic facts rather than single gold references.

Abstract

Understanding real-world videos such as movies requires integrating visual and dialogue cues to answer complex questions. Yet existing VideoQA benchmarks struggle to capture this multimodal reasoning and are largely not open-ended, given the difficulty of evaluating free-form answers. In this paper, we introduce a novel open-ended multi-modal VideoQA benchmark, MovieRecapsQA created using movie recap videos--a distinctive type of YouTube content that summarizes a film by presenting its key events through synchronized visual (recap video) and textual (recap summary) modalities. Using the recap summary, we generate K question-answer (QA) pairs (aligned with movie-subtitles) and provide the necessary "facts" needed to verify an answer in a reference-free manner. To our knowledge, this is the first open-ended VideoQA benchmark that supplies explicit textual context of the input (video and/or text); which we use for evaluation. Our benchmark provides videos of multiple lengths (i.e., recap-segments, movie-segments) and categorizations of questions (by modality and type) to enable fine-grained analysis. We evaluate the performance of seven state-of-the-art MLLMs using our benchmark and observe that: 1) visual-only questions remain the most challenging; 2) models default to textual inputs whenever available; 3) extracting factually accurate information from video content is still difficult for all models; and 4) proprietary and open-source models perform comparably on video-dependent questions.
Paper Structure (50 sections, 5 equations, 3 figures, 18 tables)

This paper contains 50 sections, 5 equations, 3 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Question Answering (VideoQA).
  4. Open-Ended Question Answering Evaluation
  5. MovieRecapsQA Benchmark
  6. MovieRecapsQA Collection & Alignment
  7. MovieRecapsQA QA Generation
  8. Statistics
  9. Evaluation Metrics
  10. Semantic Evaluation Metrics
  11. Reference-Based Evaluation Metrics
  12. Reference-Free Evaluation Metric
  13. Experiments
  14. Models
  15. Input Formats and Modalities
  16. ...and 35 more sections

Figures (3)

  • Figure 1: MovieRecapsQA Benchmark. An example from our benchmark (movie Fight Club) illustrating how MLLMs answer questions using recap-video frames and aligned movie subtitles. Q1 was constructed from Facts (3) and (5), and answering it requires integrating visual cues (Frame 2) with supporting subtitle evidence (Lines 4 and 6). We show the human answer alongside model outputs, evaluated for relevance and factuality on a 0–5 fact-grounded scale, with colors indicating quality from lowest to highest: X, X, ✓, ✓.
  • Figure 2: QA Generation Pipeline. Example question-answer pairs from recap video 6Tfmy3uGTmQ (for Liar Liar) on the recap segment "00:06:50--00:12:42" (and "00:31:44--00:50:12" from the movie). The red-highlighted text indicates the recap-segment input used to extract facts and generate the corresponding QA pair.
  • Figure S3: Alignment Between Recap Videos and Full Movies. We show alignment examples for selected videos in the dataset. Each plot maps recap-video time-stamps (y-axis) to movie time-stamps (x-axis) using our segment-shot similarity procedure. While most alignments follow a near-diagonal structure, indicating chronological correspondence, recap videos occasionally reorder scenes for narrative flow (e.g., character introductions), resulting in local misalignments.