Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpATr: MoCap 3D Human Action Recognition based on Spiral Auto-encoder and Transformer Network

Hamza Bouzid, Lahoucine Ballihi

TL;DR

A novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network), specifically designed for fixed-topology mesh sequences, which is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub.

Abstract

Recent technological advancements have significantly expanded the potential of human action recognition through harnessing the power of 3D data. This data provides a richer understanding of actions, including depth information that enables more accurate analysis of spatial and temporal characteristics. In this context, We study the challenge of 3D human action recognition.Unlike prior methods, that rely on sampling 2D depth images, skeleton points, or point clouds, often leading to substantial memory requirements and the ability to handle only short sequences, we introduce a novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network), specifically designed for fixed-topology mesh sequences. The SpATr model disentangles space and time in the mesh sequences. A lightweight auto-encoder, based on spiral convolutions, is employed to extract spatial geometrical features from each 3D mesh. These convolutions are lightweight and specifically designed for fix-topology mesh data. Subsequently, a temporal transformer, based on self-attention, captures the temporal context within the feature sequence. The self-attention mechanism enables long-range dependencies capturing and parallel processing, ensuring scalability for long sequences. The proposed method is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub, from the Archive of Motion Capture As Surface Shapes (AMASS). Our results analysis demonstrates the competitive performance of our SpATr model in 3D human action recognition while maintaining efficient memory usage. The code and the training results will soon be made publicly available at https://github.com/h-bouzid/spatr.

SpATr: MoCap 3D Human Action Recognition based on Spiral Auto-encoder and Transformer Network

TL;DR

A novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network), specifically designed for fixed-topology mesh sequences, which is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub.

Abstract

Recent technological advancements have significantly expanded the potential of human action recognition through harnessing the power of 3D data. This data provides a richer understanding of actions, including depth information that enables more accurate analysis of spatial and temporal characteristics. In this context, We study the challenge of 3D human action recognition.Unlike prior methods, that rely on sampling 2D depth images, skeleton points, or point clouds, often leading to substantial memory requirements and the ability to handle only short sequences, we introduce a novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network), specifically designed for fixed-topology mesh sequences. The SpATr model disentangles space and time in the mesh sequences. A lightweight auto-encoder, based on spiral convolutions, is employed to extract spatial geometrical features from each 3D mesh. These convolutions are lightweight and specifically designed for fix-topology mesh data. Subsequently, a temporal transformer, based on self-attention, captures the temporal context within the feature sequence. The self-attention mechanism enables long-range dependencies capturing and parallel processing, ensuring scalability for long sequences. The proposed method is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub, from the Archive of Motion Capture As Surface Shapes (AMASS). Our results analysis demonstrates the competitive performance of our SpATr model in 3D human action recognition while maintaining efficient memory usage. The code and the training results will soon be made publicly available at https://github.com/h-bouzid/spatr.
Paper Structure (16 sections, 5 equations, 5 figures, 3 tables, 2 algorithms)

This paper contains 16 sections, 5 equations, 5 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work: 3D HAR
  3. Proposed Approach:
  4. Spiral Auto-Encoder
  5. Temporal Transformer
  6. Training Algorithm
  7. Experiment
  8. Datasets
  9. Implementation Details
  10. Experimental Results
  11. Ablation Study
  12. Retraining The Model For Other Tasks
  13. Parameters Analysis
  14. Discussion
  15. Conclusion And Perspectives
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Overview of the proposed model for 3D human action recognition. Firstly, each mesh from the action sequence is mapped through the spiral auto-encoder, which extracts its embedding. The spiral auto-encoder is constructed using spiral convolution, and its architecture is illustrated in (a). Next, the sequence of embeddings is utilized by the temporal transformer, which employs self-attention to capture temporal dependencies between frame embeddings, enabling the prediction of the action class.
  • Figure 2: Examples of correct and faulty predictions.
  • Figure 3: Evolution of model accuracy based on the number of frames.
  • Figure 4: Evolution of model accuracy based on the number of multi-head self-attention heads.
  • Figure 5: confusion matrices showing the classification results of our proposed model on the test set of (a) MoVi, (b) BMLrub, and (c) Babel databases. We note that the model is trained on 24 frames.