A Survey on 3D Skeleton-Based Action Recognition Using Learning Method

TL;DR

This paper addresses the need for a comprehensive, deep-learning-oriented survey of 3D skeleton-based action recognition (3D SAR). It surveys four foundational architectures—RNNs, CNNs, GCNs, and Transformers—and presents a data-driven literature review of how skeleton data is represented and processed within each framework. It highlights NTU-RGB+D and NTU-RGB+D 120 as key benchmarks, showing that GCNs and Transformer-based methods achieve top results, with hybrid architectures offering further gains. It discusses practical challenges and future directions, including multi-architecture integration, long-horizon action recognition, and unsupervised or weakly supervised learning to handle data scarcity and annotation cost.

Abstract

3D skeleton-based action recognition (3D SAR) has gained significant attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or RGB data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3D skeleton data as a valuable modality. Subsequently, we provide a comprehensive introduction to mainstream action recognition techniques based on four fundamental deep architectures, i.e., Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Graph Convolutional Network (GCN), and Transformers. All methods with the corresponding architectures are then presented in a data-driven manner with detailed discussion. Finally, we offer insights into the current largest 3D skeleton dataset, NTU-RGB+D, and its new edition, NTU-RGB+D 120, along with an overview of several top-performing algorithms on these datasets. To the best of our knowledge, this research represents the first comprehensive discussion of deep learning-based action recognition using 3D skeleton data.

Figures (6)

• Figure 1: Examples of skeleton data in NTU RGB+D / NTU RGB+D 120 datasets shahroudy2016ntuliu2019ntu. (a) is the Configuration of 25 body joints in the dataset. (b) illustrates RGB+joints representation of the human body.
• Figure 2: The general pipeline of skeleton-based action recognition using deep learning methods. Firstly, the skeleton data was obtained in two ways, directly from depth sensors or from pose estimation algorithms. The skeleton will be sent into RNNs, CNNs, GCNs, or Transformer-based neural networks. Finally, we get the accurate action category.
• Figure 3: Examples of mentioned methods for dealing with spatial modeling problems. (a) shows a two-stream framework that enhances the spatial information by adding a new stream wang2017modeling. (b) illustrates a data-driven technique that addresses the spatial modeling ability by giving a transform towards original skeleton sequence data liu2016spatio.
• Figure 4: Data-driven based method. (a) shows the different importance among different joints for a given skeleton action liu2017global. (b) give shows the feature representation processes, from left to right are original input skeleton frames, transformed input frames, and extracted salient motion features respectively lee2017ensemble.
• Figure 5: Examples of the proposed representation skeleton image. (a) shows Skeleton sequence shape-motion representations li2019learning generated from "pick up with one hand" on Northwestern-UCLA datasetwang2014cross (b) shows the SkeleMotion representation workflow caetano2019skeleton.