Movie Description

TL;DR

This work introduces the Large Scale Movie Description Challenge (LSMDC), a large, open-domain dataset of transcribed Audio Descriptions (ADs) aligned to full-length movies, paired with script data for comparison. It benchmarks two main description pipelines: a Semantic Parsing + SMT approach that converts visual input into a semantic representation and translates it into sentences, and a Visual-Labels + LSTM approach that learns robust visual classifiers (verbs, objects, places) and uses an LSTM to generate descriptions. Across MPII-MD and MVAD datasets, ADs prove more visual and accurately describe visible events than scripts, with Visual-Labels delivering strong automatic METEOR scores and favorable human judgments, though long-tail, multi-sentence movie descriptions remain challenging. The LSMDC benchmark, evaluation server, and public/blind test splits enable rigorous cross-dataset comparisons and drive progress toward accessible, automatic movie understanding. The work also analyzes dataset properties, evaluation metrics, and the long-tail distribution of descriptions, highlighting both the potential and remaining hurdles for production-grade, visually grounded video description.

Abstract

Audio Description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. In total the Large Scale Movie Description Challenge (LSMDC) contains a parallel corpus of 118,114 sentences and video clips from 202 movies. First we characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing ADs to scripts, we find that ADs are indeed more visual and describe precisely what is shown rather than what should happen according to the scripts created prior to movie production. Furthermore, we present and compare the results of several teams who participated in a challenge organized in the context of the workshop "Describing and Understanding Video & The Large Scale Movie Description Challenge (LSMDC)", at ICCV 2015.

Figures (11)

• Figure 1: Audio description (AD) and movie script samples from the movie "Ugly Truth".
• Figure 2: Audio description (AD) and movie script samples from the movies "Harry Potter and the Prisoner of Azkaban", "This is 40", and "Les Miserables". Typical mistakes contained in scripts marked in red italic.
• Figure 3: Some of the diverse verbs / actions present in our Large Scale Movie Description Challenge (LSMDC).
• Figure 4: AD dataset collection. From the movie "Life of Pi". Line 2 and 3: Vocal isolation of movie and AD soundtrack. Second and third rows shows movie and AD audio signals after voice isolation. The two circles show the AD segments on the AD mono channel track. A pause (flat signal) between two AD narration parts shows the natural AD narration segmentation while the narrator stops and then continues describing the movie. We automatically segment AD audio based on these natural pauses. At first row, you can also see the transcription related to first and second AD narration parts on top of second and third image shots.
• Figure 5: Overview of our movie description approaches: (a) SMT-based approach, adapted from rohrbach13iccv; (b) our proposed LSTM-based approach.