Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human Stone Toolmaking Action Grammar (HSTAG): A Challenging Benchmark for Fine-grained Motor Behavior Recognition

Cheng Liu, Xuyang Yan, Zekun Zhang, Cheng Ding, Tianhao Zhao, Shaya Jannati, Cynthia Martinez, Dietrich Stout

TL;DR

The paper introduces HSTAG, a domain-specific video dataset for fine-grained human action recognition in stone toolmaking, comprising 18,739 clipped videos across seven action classes and featuring multi-view capture, tool switching, and rapid action transitions. It details data collection from experts, a BORIS-based annotation pipeline with a seven-action grammar, and a 75:25 train/test split highlighting class imbalances. The authors benchmark three SOTA models (VideoMAEv2, TimeSformer, ResNet+GRU), analyze performance and class separability, and discuss challenges such as high-frequency frames and action similarity. The work positions HSTAG as a challenging benchmark to test generalizability of CV techniques in rarely seen domains and to spur development of methods robust to unbalanced temporal sequences.

Abstract

Action recognition has witnessed the development of a growing number of novel algorithms and datasets in the past decade. However, the majority of public benchmarks were constructed around activities of daily living and annotated at a rather coarse-grained level, which lacks diversity in domain-specific datasets, especially for rarely seen domains. In this paper, we introduced Human Stone Toolmaking Action Grammar (HSTAG), a meticulously annotated video dataset showcasing previously undocumented stone toolmaking behaviors, which can be used for investigating the applications of advanced artificial intelligence techniques in understanding a rapid succession of complex interactions between two hand-held objects. HSTAG consists of 18,739 video clips that record 4.5 hours of experts' activities in stone toolmaking. Its unique features include (i) brief action durations and frequent transitions, mirroring the rapid changes inherent in many motor behaviors; (ii) multiple angles of view and switches among multiple tools, increasing intra-class variability; (iii) unbalanced class distributions and high similarity among different action sequences, adding difficulty in capturing distinct patterns for each action. Several mainstream action recognition models are used to conduct experimental analysis, which showcases the challenges and uniqueness of HSTAG https://nyu.databrary.org/volume/1697.

Human Stone Toolmaking Action Grammar (HSTAG): A Challenging Benchmark for Fine-grained Motor Behavior Recognition

TL;DR

The paper introduces HSTAG, a domain-specific video dataset for fine-grained human action recognition in stone toolmaking, comprising 18,739 clipped videos across seven action classes and featuring multi-view capture, tool switching, and rapid action transitions. It details data collection from experts, a BORIS-based annotation pipeline with a seven-action grammar, and a 75:25 train/test split highlighting class imbalances. The authors benchmark three SOTA models (VideoMAEv2, TimeSformer, ResNet+GRU), analyze performance and class separability, and discuss challenges such as high-frequency frames and action similarity. The work positions HSTAG as a challenging benchmark to test generalizability of CV techniques in rarely seen domains and to spur development of methods robust to unbalanced temporal sequences.

Abstract

Action recognition has witnessed the development of a growing number of novel algorithms and datasets in the past decade. However, the majority of public benchmarks were constructed around activities of daily living and annotated at a rather coarse-grained level, which lacks diversity in domain-specific datasets, especially for rarely seen domains. In this paper, we introduced Human Stone Toolmaking Action Grammar (HSTAG), a meticulously annotated video dataset showcasing previously undocumented stone toolmaking behaviors, which can be used for investigating the applications of advanced artificial intelligence techniques in understanding a rapid succession of complex interactions between two hand-held objects. HSTAG consists of 18,739 video clips that record 4.5 hours of experts' activities in stone toolmaking. Its unique features include (i) brief action durations and frequent transitions, mirroring the rapid changes inherent in many motor behaviors; (ii) multiple angles of view and switches among multiple tools, increasing intra-class variability; (iii) unbalanced class distributions and high similarity among different action sequences, adding difficulty in capturing distinct patterns for each action. Several mainstream action recognition models are used to conduct experimental analysis, which showcases the challenges and uniqueness of HSTAG https://nyu.databrary.org/volume/1697.

Paper Structure

This paper contains 13 sections, 3 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Human Stone Toolmaking Action Grammar dataset
  4. Data collection
  5. Data annotation
  6. Dataset statistics
  7. Experiments analysis
  8. Benchmark models
  9. Experimental setup
  10. Evaluation metrics
  11. Result discussions
  12. Analysis on class separability
  13. Conclusion

Figures (4)

  • Figure 1: Stone toolmaking shot from the top/ego view (A) and the front view (B).
  • Figure 2: Samples for each of the 7 actions in HSTAG. We use red rectangles to highlight the participant's slight hand movement.
  • Figure 3: The user interface of the BORIS annotation tool.
  • Figure 4: Two-dimensional visualization on the embeddings learned from the trained models on HSTAG.