Dense-Captioning Events in Videos

TL;DR

Dense-captioning events tackles the problem of detecting multiple temporally localized events in videos and describing each with natural language. The authors present a unified model that combines a multi-scale event proposal module (variant of DAPs) with a context-aware captioning module that attends to other events in the video, enabling descriptions that reflect inter-event dependencies. They introduce ActivityNet Captions, a large-scale open-domain dataset with long videos, overlapping events, and 100k sentences to benchmark dense-captioning, retrieval, and localization. Across experiments, incorporating temporal context improves caption quality and retrieval/localization performance, demonstrating the value of modeling event interdependencies in videos.

Abstract

Most natural videos contain numerous events. For example, in a video of a "man playing a piano", the video might also contain "another man dancing" or "a crowd clapping". We introduce the task of dense-captioning events, which involves both detecting and describing events in a video. We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language. Our model introduces a variant of an existing proposal module that is designed to capture both short as well as long events that span minutes. To capture the dependencies between the events in a video, our model introduces a new captioning module that uses contextual information from past and future events to jointly describe all events. We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events. ActivityNet Captions contains 20k videos amounting to 849 video hours with 100k total descriptions, each with it's unique start and end time. Finally, we report performances of our model for dense-captioning events, video retrieval and localization.

Figures (16)

• Figure 1: Dense-captioning events in a video involves detecting multiple events that occur in a video and describing each event using natural language. These events are temporally localized in the video with independent start and end times, resulting in some events that might also occur concurrently and overlap in time.
• Figure 2: Complete pipeline for dense-captioning events in videos with descriptions. We first extract C3D features from the input video. These features are fed into our proposal module at varying stride to predict both short as well as long events. Each proposal, which consists of a unique start and end time and a hidden representation, is then used as input into the captioning module. Finally, this captioning model leverages context from neighboring events to generate each event description.
• Figure 3: The parts of speech distribution of ActivityNet Captions compared with Visual Genome, a dataset with multiple sentence annotations per image. There are many more verbs and pronouns represented in ActivityNet Captions, as the descriptions often focus on actions.
• Figure 4: Qualitative dense-captioning captions generated using our model. We show captions with the highest overlap with ground truth captions.
• Figure 5: Evaluating our proposal module, we find that sampling videos at varying strides does in fact improve the module's ability to localize events, specially longer events.