Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding

Yidan Sun, Jianfei Yu, Boyang Li

TL;DR

A large-scale multilingual video story dataset named Multilingual Synopses of Movie Narratives (M-SYMON), containing 13,166 movie summary videos from 7 languages, as well as manual annotation of fine-grained video-text correspondences for 101.5 hours of video, demonstrating the effectiveness of the annotations.

Abstract

Story video-text alignment, a core task in computational story understanding, aims to align video clips with corresponding sentences in their descriptions. However, progress on the task has been held back by the scarcity of manually annotated video-text correspondence and the heavy concentration on English narrations of Hollywood movies. To address these issues, in this paper, we construct a large-scale multilingual video story dataset named Multilingual Synopses of Movie Narratives (M-SYMON), containing 13,166 movie summary videos from 7 languages, as well as manual annotation of fine-grained video-text correspondences for 101.5 hours of video. Training on the human annotated data from SyMoN outperforms the SOTA methods by 15.7 and 16.2 percentage points on Clip Accuracy and Sentence IoU scores, respectively, demonstrating the effectiveness of the annotations. As benchmarks for future research, we create 6 baseline approaches with different multilingual training strategies, compare their performance in both intra-lingual and cross-lingual setups, exemplifying the challenges of multilingual video-text alignment. The dataset is released at: https://github.com/insundaycathy/M-SyMoN

Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding

TL;DR

A large-scale multilingual video story dataset named Multilingual Synopses of Movie Narratives (M-SYMON), containing 13,166 movie summary videos from 7 languages, as well as manual annotation of fine-grained video-text correspondences for 101.5 hours of video, demonstrating the effectiveness of the annotations.

Abstract

Story video-text alignment, a core task in computational story understanding, aims to align video clips with corresponding sentences in their descriptions. However, progress on the task has been held back by the scarcity of manually annotated video-text correspondence and the heavy concentration on English narrations of Hollywood movies. To address these issues, in this paper, we construct a large-scale multilingual video story dataset named Multilingual Synopses of Movie Narratives (M-SYMON), containing 13,166 movie summary videos from 7 languages, as well as manual annotation of fine-grained video-text correspondences for 101.5 hours of video. Training on the human annotated data from SyMoN outperforms the SOTA methods by 15.7 and 16.2 percentage points on Clip Accuracy and Sentence IoU scores, respectively, demonstrating the effectiveness of the annotations. As benchmarks for future research, we create 6 baseline approaches with different multilingual training strategies, compare their performance in both intra-lingual and cross-lingual setups, exemplifying the challenges of multilingual video-text alignment. The dataset is released at: https://github.com/insundaycathy/M-SyMoN
Paper Structure (43 sections, 3 equations, 9 figures, 16 tables)

This paper contains 43 sections, 3 equations, 9 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Story Video-Text Alignment
  4. Movie Story Datasets
  5. Multilingual Story Understanding
  6. Task Formulation
  7. Dataset Construction
  8. Data Source and Statistics
  9. Human Annotation
  10. Data Preprocessing
  11. Dataset Split
  12. Dataset Analysis
  13. Methodology
  14. Base Model
  15. Multilingual Finetuning Methods
  16. ...and 28 more sections

Figures (9)

  • Figure 1: An example alignment between a video clip sequence and a sentence sequence from M-SyMoN. One text chuck may correspond to several video clips, while some video clips may not match any textual description. The example is from https://youtu.be/n5v9hzSYxPQ, a summary of the movie 407 Dark Flight (2012)
  • Figure 2: The top 5 countries of release for movies recapped in each language.
  • Figure 3: The top 5 themes for movies recapped in each language.
  • Figure 4: Cross-lingual transfer results of CCLM-two-stage based on F1 score. The language names are abbreviated as: English ="en", Chinese = "ch", Spanish = "es", French = "fr", Portuguese "pt", Hindi = "hi", Russian = "ru".
  • Figure 5: The top 5 genres for movies recapped in each language
  • ...and 4 more figures