Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Movie101v2: Improved Movie Narration Benchmark

Zihao Yue, Yepeng Zhang, Ziheng Wang, Qin Jin

TL;DR

This work introduces Movie101v2, a large-scale, bilingual dataset with enhanced data quality specifically designed for movie narration, and proposes breaking down the ultimate goal of automatic movie narration into three progressive stages, offering a clear roadmap with corresponding evaluation metrics.

Abstract

Automatic movie narration aims to generate video-aligned plot descriptions to assist visually impaired audiences. Unlike standard video captioning, it involves not only describing key visual details but also inferring plots that unfold across multiple movie shots, presenting distinct and complex challenges. To advance this field, we introduce Movie101v2, a large-scale, bilingual dataset with enhanced data quality specifically designed for movie narration. Revisiting the task, we propose breaking down the ultimate goal of automatic movie narration into three progressive stages, offering a clear roadmap with corresponding evaluation metrics. Based on our new benchmark, we baseline a range of large vision-language models, including GPT-4V, and conduct an in-depth analysis of the challenges in narration generation. Our findings highlight that achieving applicable movie narration generation is a fascinating goal that requires significant research.

Movie101v2: Improved Movie Narration Benchmark

TL;DR

This work introduces Movie101v2, a large-scale, bilingual dataset with enhanced data quality specifically designed for movie narration, and proposes breaking down the ultimate goal of automatic movie narration into three progressive stages, offering a clear roadmap with corresponding evaluation metrics.

Abstract

Automatic movie narration aims to generate video-aligned plot descriptions to assist visually impaired audiences. Unlike standard video captioning, it involves not only describing key visual details but also inferring plots that unfold across multiple movie shots, presenting distinct and complex challenges. To advance this field, we introduce Movie101v2, a large-scale, bilingual dataset with enhanced data quality specifically designed for movie narration. Revisiting the task, we propose breaking down the ultimate goal of automatic movie narration into three progressive stages, offering a clear roadmap with corresponding evaluation metrics. Based on our new benchmark, we baseline a range of large vision-language models, including GPT-4V, and conduct an in-depth analysis of the challenges in narration generation. Our findings highlight that achieving applicable movie narration generation is a fascinating goal that requires significant research.
Paper Structure (23 sections, 2 equations, 13 figures, 4 tables)

This paper contains 23 sections, 2 equations, 13 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Data: Movie101v2
  3. Scaling Up
  4. Quality Enhancement
  5. Data Statistics
  6. Task: Movie Narration Generation
  7. Task Roadmap
  8. Evaluation
  9. Method: Benchmarking and Analysis
  10. Benchmarking
  11. Analysis
  12. Visual Perception
  13. Textual Generation
  14. Related Works
  15. Conclusion
  16. ...and 8 more sections

Figures (13)

  • Figure 1: Examples from other datasets (left) and Movie101v2 (right) where cases are from Goodbye Mr. Loser.
  • Figure 2: Distribution of character counts in movie casts (left) and narration paragraphs (right).
  • Figure 3: Distribution of narration text length and movie clip duration in Movie101v2-zh.
  • Figure 4: Human evaluation results on the necessity of various multi-modal contextual information for L3 narrations. The top section displays the percentage of "Yes" responses for whether a clip can be accurately narrated without corresponding context. The bottom section shows the "Yes" percentage with all context information being removed progressively.
  • Figure 5: Model performance on Movie101v2. All L1/L2 scores are rescaled to a range of 0-100.
  • ...and 8 more figures