Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Scale Spatial-Temporal Self-Attention Graph Convolutional Networks for Skeleton-based Action Recognition

Ikuo Nakamura

TL;DR

The paper tackles skeleton-based action recognition by learning context-dependent, adaptive topologies for joints within a GCN framework. It introduces MSST-GCN, which combines spatial self-attention with an adaptive topology $ ilde{A}$ and temporal self-attention, followed by multi-scale dilated convolutions to capture long-range spatial and temporal dependencies. Two streams produce complementary representations that are fused via a learnable channel-wise weight and trained with a loss that includes cross-entropy and MMD terms. Experiments on SHREC'17, NTU-RGB+D 60, and NW-UCLA demonstrate strong performance and robustness across benchmarks, with competitive or state-of-the-art results and flexible ensemble capabilities.

Abstract

Skeleton-based gesture recognition methods have achieved high success using Graph Convolutional Network (GCN). In addition, context-dependent adaptive topology as a neighborhood vertex information and attention mechanism leverages a model to better represent actions. In this paper, we propose self-attention GCN hybrid model, Multi-Scale Spatial-Temporal self-attention (MSST)-GCN to effectively improve modeling ability to achieve state-of-the-art results on several datasets. We utilize spatial self-attention module with adaptive topology to understand intra-frame interactions within a frame among different body parts, and temporal self-attention module to examine correlations between frames of a node. These two are followed by multi-scale convolution network with dilations, which not only captures the long-range temporal dependencies of joints but also the long-range spatial dependencies (i.e., long-distance dependencies) of node temporal behaviors. They are combined into high-level spatial-temporal representations and output the predicted action with the softmax classifier.

Multi-Scale Spatial-Temporal Self-Attention Graph Convolutional Networks for Skeleton-based Action Recognition

TL;DR

The paper tackles skeleton-based action recognition by learning context-dependent, adaptive topologies for joints within a GCN framework. It introduces MSST-GCN, which combines spatial self-attention with an adaptive topology and temporal self-attention, followed by multi-scale dilated convolutions to capture long-range spatial and temporal dependencies. Two streams produce complementary representations that are fused via a learnable channel-wise weight and trained with a loss that includes cross-entropy and MMD terms. Experiments on SHREC'17, NTU-RGB+D 60, and NW-UCLA demonstrate strong performance and robustness across benchmarks, with competitive or state-of-the-art results and flexible ensemble capabilities.

Abstract

Skeleton-based gesture recognition methods have achieved high success using Graph Convolutional Network (GCN). In addition, context-dependent adaptive topology as a neighborhood vertex information and attention mechanism leverages a model to better represent actions. In this paper, we propose self-attention GCN hybrid model, Multi-Scale Spatial-Temporal self-attention (MSST)-GCN to effectively improve modeling ability to achieve state-of-the-art results on several datasets. We utilize spatial self-attention module with adaptive topology to understand intra-frame interactions within a frame among different body parts, and temporal self-attention module to examine correlations between frames of a node. These two are followed by multi-scale convolution network with dilations, which not only captures the long-range temporal dependencies of joints but also the long-range spatial dependencies (i.e., long-distance dependencies) of node temporal behaviors. They are combined into high-level spatial-temporal representations and output the predicted action with the softmax classifier.
Paper Structure (14 sections, 4 equations, 3 figures, 6 tables)

This paper contains 14 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Self-attentions
  5. Multi-Scale Convolution
  6. Two stream networks
  7. Multi-Scale Spatial-Temporal self-attention GCN
  8. Ensemble with Multi-modal representation
  9. Experiment
  10. Datasets
  11. Experimental settings
  12. Experimental results
  13. Ablation studies
  14. Conclusion

Figures (3)

  • Figure 1: SSA-GC + MS-TC
  • Figure 2: TSA + MS-SC
  • Figure 4: The overall framework of our MSST-GCN, where $SSAm\in \mathbb{R}^{N \times N}$ represents spatial self-attention map in SSA and $TSAm \in \mathbb{R}^{T \times T}$ represents temporal self-attention map in TSA. Three multi-head attentions are described in the figure.