Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MM-SEAL: A Large-scale Video Dataset of Multi-person Multi-grained Spatio-temporally Action Localization

Shimin Chen, Wei Li, Chen Chen, Jianyang Gu, Jiaming Chu, Xunqiang Tao, Yandong Guo

TL;DR

The first to propose a new benchmark for multi-person spatio-temporal complex activity localization, where complex semantic and long duration bring new challenges to localization tasks, and finds that atomic action features can improve the complex activity localization performance.

Abstract

In this paper, we introduce a novel large-scale video dataset dubbed MM-SEAL for multi-person multi-grained spatio-temporal action localization among human daily life. We are the first to propose a new benchmark for multi-person spatio-temporal complex activity localization, where complex semantic and long duration bring new challenges to localization tasks. We observe that limited atomic actions can be combined into many complex activities. MM-SEAL provides both atomic action and complex activity annotations, producing 111.7k atomic actions spanning 172 action categories and 17.7k complex activities spanning 200 activity categories. We explore the relationship between atomic actions and complex activities, finding that atomic action features can improve the complex activity localization performance. Also, we propose a new network which generates temporal proposals and labels simultaneously, termed Faster-TAD. Finally, our evaluations show that visual features pretrained on MM-SEAL can improve the performance on other action localization benchmarks. We will release the dataset and the project code upon publication of the paper.

MM-SEAL: A Large-scale Video Dataset of Multi-person Multi-grained Spatio-temporally Action Localization

TL;DR

The first to propose a new benchmark for multi-person spatio-temporal complex activity localization, where complex semantic and long duration bring new challenges to localization tasks, and finds that atomic action features can improve the complex activity localization performance.

Abstract

In this paper, we introduce a novel large-scale video dataset dubbed MM-SEAL for multi-person multi-grained spatio-temporal action localization among human daily life. We are the first to propose a new benchmark for multi-person spatio-temporal complex activity localization, where complex semantic and long duration bring new challenges to localization tasks. We observe that limited atomic actions can be combined into many complex activities. MM-SEAL provides both atomic action and complex activity annotations, producing 111.7k atomic actions spanning 172 action categories and 17.7k complex activities spanning 200 activity categories. We explore the relationship between atomic actions and complex activities, finding that atomic action features can improve the complex activity localization performance. Also, we propose a new network which generates temporal proposals and labels simultaneously, termed Faster-TAD. Finally, our evaluations show that visual features pretrained on MM-SEAL can improve the performance on other action localization benchmarks. We will release the dataset and the project code upon publication of the paper.
Paper Structure (26 sections, 5 figures, 6 tables)

This paper contains 26 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Spatio-temporal Action Localization Datasets.
  4. Spatio-temporal action detection method
  5. The MM-SEAL Dataset
  6. Data Collection
  7. Category Selection.
  8. Data Selection.
  9. Subject Selection.
  10. Data Annotation
  11. Spatial Annotation.
  12. Temporal and Semantic Annotation.
  13. Quality Assurance.
  14. Statistics
  15. Characteristics and Challenges
  16. ...and 11 more sections

Figures (5)

  • Figure 1: The overview of MM-SEAL dataset. With the help of multi-object tracking, person re-identification, and annotator correction, we obtain tubelets for each subject who are related to the activity. Then we provide complex activities instances and atomic actions instances of each tubelet. In complex activity level, we annotate instances from 200 classes, such as "sumo" marked in green. In atomic action level, we choose atomic actions from the action vocabulary(bottom).
  • Figure 2: The distribution of instance duration(a)(b) and instance numbers in each video(c)(d) for action instances. The abscissa in (a)(b) represents mean the duration of action in seconds, and that in (c)(d) represents the instance numbers in each video. The ordinate represents the instance numbers to the abscissa. AA denotes the Atomic Action; CA denotes the Complex Activity.
  • Figure 3: Proximity-Category Proposal Block. The first row shows the ground truth segments. The second row is coarse proposals from Proposal Generation Mechanism. The last row shows that the proposal with unsatisfied IoU will be set to a Proximity-Category according to its nearby ground truth segment. For example, proposal 2 has a label of "using the rowing machine - proximity’’.
  • Figure 4: Proposal Attention Module. Proposal features are generated from proposal generation outputs and the shared features by a ROI layer. Then, encoder layer is followed to further encode the proposal representation. Finally, Self and Cross Attention block is applied to model the proposal semantic features.
  • Figure 5: Auxiliary-Feature Block. Two streams of features go through base module respectively. Then, they are combined along the temporal dimension. The rest of the network keeps the same.