Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research

Atousa Torabi, Christopher Pal, Hugo Larochelle, Aaron Courville

TL;DR

Addresses the lack of large-scale, open-domain video descriptions by leveraging DVS narrations embedded in DVDs. Introduces a semi-automatic pipeline that isolates DVS audio, aligns it to video, and transcribes it to produce the Montreal Video Annotation Dataset (M-VAD). The dataset comprises 84.6 hours across 92 DVDs with 55,904 sentences and POS-tagged vocabulary, and includes a balanced train/validation/test split. Preliminary results with an LSTM-based description approach show semantically meaningful captions that reflect the visual content, underscoring the dataset's value for training modern video-description models.

Abstract

In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.

Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research

TL;DR

Addresses the lack of large-scale, open-domain video descriptions by leveraging DVS narrations embedded in DVDs. Introduces a semi-automatic pipeline that isolates DVS audio, aligns it to video, and transcribes it to produce the Montreal Video Annotation Dataset (M-VAD). The dataset comprises 84.6 hours across 92 DVDs with 55,904 sentences and POS-tagged vocabulary, and includes a balanced train/validation/test split. Preliminary results with an LSTM-based description approach show semantically meaningful captions that reflect the visual content, underscoring the dataset's value for training modern video-description models.

Abstract

In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.

Paper Structure

This paper contains 8 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DVS-Derived Dataset Collection
  4. DVS Narrations Segmentation Using Vocal Isolation
  5. Movie/DVS Alignment And Professional Transcription
  6. DVS-Derived Dataset Statistics and Comparison With Other Datasets
  7. DVS-Derived Dataset Corpus Preparation
  8. Conclusion

Figures (3)

  • Figure 1: DVS dataset collection. From the movie "Life of the Pie". Line 2 and 3: Vocal isolation of movie and DVS soundtrack. Second and third rows shows movie and DVS audio signals after voice isolation. The two circles show the DVS segments on the DVS mono channel track. A pause (flat signal) between two DVS narration parts shows the natural DVS narration segmentation while the narrator stops and then continues describing the movie. We automatically segment DVS audio based on these natural pauses. At first row, you can also see the transcription related to first and second DVS narration parts on top of second and third image shots.
  • Figure 2: (a) 40 most frequent verbs and (b) 40 most frequent nouns.
  • Figure 3: Four samples of generated sentences by our LSTM model and DVS narrations from the movies: A: Charile st. cloud, B: How do you know, C: The roommate, and D: The big year.