Moments in Time Dataset: one million videos for event understanding

TL;DR

Moments in Time tackles the problem of understanding dynamic events in short videos by providing a large-scale corpus of $3$-second clips labeled with a vocabulary of $339$ verbs. The authors combine a data-collection pipeline from multiple sources with Amazon Mechanical Turk-based annotation and multi-modal baselines spanning spatial, temporal, and auditory cues. Key contributions include the vocabulary design, annotation protocol with consensus thresholds, dataset statistics, and baseline results showing the benefits of multi-modal fusion as well as cross-dataset transfer. The dataset is intended to drive the development of models capable of scalable, abstract reasoning about events across varied agents and environments.

Abstract

We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time ("opening" is "closing" in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.

Figures (7)

• Figure 1: Sample Videos. Day-to-day events can happen to many types of actors, in different environments, and at different scales. Moments in Time dataset has a significant intra-class variation among the categories. Here we illustrate one frame for a few video samples and actions. For example, car engines, books, and tulips can all open.
• Figure 2: User interface. An example for our binary annotation task for the action cooking.
• Figure 3: Dataset Statistics.Left: Distribution of the number of videos belonging to each category. Middle: Per class distribution of videos that have humans, animals, or objects as agents completing actions. Right: Per class distribution of videos that require audio to recognize the class category and videos that can be categorized with only visual information.
• Figure 4: Comparison to Datasets. For each dataset we provide different comparisons. Left: the total number of action labels in the training set. Middle: the average number of videos per class (some videos can belong to multiple classes).Right: the coverage of objects and scenes recognized (top 1) by networks trained on Places and Imagenet.
• Figure 5: Overview of top detections for several single stream models. The ground truth label and top three model predictions are listed for representative frames of videos.