Action100M: A Large-scale Video Action Dataset

TL;DR

Action100M addresses the need for large-scale open-domain video action data by constructing a 100M-action dataset from 1.2 million instructional videos using an automated pipeline that couples hierarchical temporal segmentation, a Tree-of-Captions for multi-level captions, and an LLM-based Self-Refine reasoning step. The core contribution is a scalable framework that yields rich, structured annotations across broad domains, enabling robust open-vocabulary action recognition and world modeling. Empirically, pretraining VL-JEPA on Action100M delivers consistent scaling benefits and strong zero-shot transfer across diverse benchmarks, with semantic resampling further boosting sample efficiency. This work provides a foundation for scalable video understanding, action anticipation, and long-horizon planning in embodied AI systems.

Abstract

Inferring physical actions from visual observations is a fundamental capability for advancing machine intelligence in the physical world. Achieving this requires large-scale, open-vocabulary video action datasets that span broad domains. We introduce Action100M, a large-scale dataset constructed from 1.2M Internet instructional videos (14.6 years of duration), yielding O(100 million) temporally localized segments with open-vocabulary action supervision and rich captions. Action100M is generated by a fully automated pipeline that (i) performs hierarchical temporal segmentation using V-JEPA 2 embeddings, (ii) produces multi-level frame and segment captions organized as a Tree-of-Captions, and (iii) aggregates evidence with a reasoning model (GPT-OSS-120B) under a multi-round Self-Refine procedure to output structured annotations (brief/detailed action, actor, brief/detailed caption). Training VL-JEPA on Action100M demonstrates consistent data-scaling improvements and strong zero-shot performance across diverse action recognition benchmarks, establishing Action100M as a new foundation for scalable research in video understanding and world modeling.

Figures (13)

• Figure 1: Scaling on Action100M improves zero-shot action recognition consistently.
• Figure 2: Action100M Data Pipeline. Our pipeline first applies hierarchical temporal segmentation to decompose the video into semantically coherent segments at multiple temporal scales. For each segment, we generate video caption and frame captions, capturing both temporal and spatial information. Next, we prompt LLM to aggregate the captions, extracting final annotations.
• Figure 3: Example of hierarchical structure in Action100M annotations (with brief action description labels shown). Source video: https://www.youtube.com/watch?v=NYRlBWgLbKU. Brief caption of the entire video: A woman roasts almonds, blends them into butter, and pours the butter into a jar. Detailed caption: The video opens with the presenter in a bright kitchen speaking to the camera. She spreads raw almonds on a parchment‑lined tray, places the tray in a pre‑heated 350 °F oven, and after roasting lets the nuts cool to room temperature. She then transfers the almonds to a Vitamix blender, removes the lid, inserts a tamper, and sets the machine on high. While the blender runs she presses the almonds down with the tamper, first creating a fine flour and then a thick creamy butter within about one minute. She pours the almond butter into a clear storage jar, scoops it with a large wooden spoon and stirs it to smooth the surface, then concludes the segment with a brief thank‑you.
• Figure 4: Statistics of Action100M source videos and metadata. Distributions of (left to right) video upload year, view count, video duration, and transcript length, computed over the subset of videos for which metadata is available.
• Figure 5: Word cloud of video titles in Action100M. Frequently occurring words reflect the instructional and procedural nature of the dataset, with dominant terms related to cooking, DIY activities, and everyday physical tasks.