IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition

TL;DR

This work proposes a novel transformer-based network (IIP-Transformer) which incorporates joint-level (intra-part) and part-level (inter-part) interactions simultaneously is the keypoint for IIP-Transformer to fully exploit part-level data, making considerable improvements in both coarse-grained and fine-grained action recognition.

Abstract

Recently, Transformer-based networks have shown great promise on skeleton-based action recognition tasks. The ability to capture global and local dependencies is the key to success while it also brings quadratic computation and memory cost. Another problem is that previous studies mainly focus on the relationships among individual joints, which often suffers from the noisy skeleton joints introduced by the noisy inputs of sensors or inaccurate estimations. To address the above issues, we propose a novel Transformer-based network (IIP-Transformer). Instead of exploiting interactions among individual joints, our IIP-Transformer incorporates body joints and parts interactions simultaneously and thus can capture both joint-level (intra-part) and part-level (inter-part) dependencies efficiently and effectively. From the data aspect, we introduce a part-level skeleton data encoding that significantly reduces the computational complexity and is more robust to joint-level skeleton noise. Besides, a new part-level data augmentation is proposed to improve the performance of the model. On two large-scale datasets, NTU-RGB+D 60 and NTU RGB+D 120, the proposed IIP-Transformer achieves the-state-of-art performance with more than 8x less computational complexity than DSTA-Net, which is the SOTA Transformer-based method.

Figures (8)

• Figure 1: Illustration of our main idea. The body joints are divided into 5 parts. The Inter-Part branch is used to explore relationships between parts and the Intra-Part branch aims to capture dependencies between joints in the same part.
• Figure 2: The overall architecture of the proposed pipeline which is composed of Partition Encoding and IIP-Transformer.
• Figure 3: Illustration of the Partition Encoding Procedure.
• Figure 4: Illustration of the IIPA mechanism. The original multi-head self attention is used to explore relations between tokens while the function $f_{intra}$ is used to capture internal relations of a token.
• Figure 5: Two Transformer structures. (A) Flatten the spatial-temporal data into a single sequence and use an standard transformer. (B) Explore spatial and temporal relations with S-IIPA and T-IIPA respectively and fuse them by a feed-forward layer.