Human Action Recognition from Point Clouds over Time

TL;DR

The paper addresses human action recognition from dense 3D point clouds, tackling limitations of traditional video-based and skeletal approaches by leveraging depth sensors and monocular depth estimation to produce rich 3D representations.It introduces a two-scenario pipeline for obtaining human point clouds (depth/IR or RGB with monocular depth estimation), followed by instance/body parsing, tracking, and robust point sampling to feed a novel 3D backbone that fuses point-based embeddings with sparse convolution in a 4D spatio-temporal grid.Key contributions include the T-Net embedding module, a 4D sparse convolutional backbone with MS-TCN, and comprehensive ablations demonstrating gains from auxiliary features (normals, part labels, infrared), leading to competitive results on NTU RGB-D 120; ensemble of depth-based and estimated-depth inputs yields a new high in cross-subject accuracy.The work demonstrates that dense 3D representations and hybrid point-voxel architectures can achieve strong HAR performance in realistic settings, with potential practical impact for surveillance, sports analytics, and autonomous systems.

Abstract

Recent research into human action recognition (HAR) has focused predominantly on skeletal action recognition and video-based methods. With the increasing availability of consumer-grade depth sensors and Lidar instruments, there is a growing opportunity to leverage dense 3D data for action recognition, to develop a third way. This paper presents a novel approach for recognizing actions from 3D videos by introducing a pipeline that segments human point clouds from the background of a scene, tracks individuals over time, and performs body part segmentation. The method supports point clouds from both depth sensors and monocular depth estimation. At the core of the proposed HAR framework is a novel backbone for 3D action recognition, which combines point-based techniques with sparse convolutional networks applied to voxel-mapped point cloud sequences. Experiments incorporate auxiliary point features including surface normals, color, infrared intensity, and body part parsing labels, to enhance recognition accuracy. Evaluation on the NTU RGB- D 120 dataset demonstrates that the method is competitive with existing skeletal action recognition algorithms. Moreover, combining both sensor-based and estimated depth inputs in an ensemble setup, this approach achieves 89.3% accuracy when different human subjects are considered for training and testing, outperforming previous point cloud action recognition methods.

Figures (5)

• Figure : Figure 1: An overview of the proposed pipeline for obtaining human point clouds over time from aligned depth/IR pairs. Instance and body part segmentation is performed at each frame, followed by mask denoising in image space. Tracking is then applied to bounding boxes fit to the instances, followed by projection to 3D using the depth images, and denoising in 3D space. Note that body part labels are shown in the middle column.
• Figure : Figure 2: The pipeline for obtaining denoised human point clouds over time from RGB videos using monocular depth estimation, with body part labels displayed in the middle column.
• Figure : Figure 3: The proposed model for action recognition from human point clouds over time, where $T$ is the number of frames, $N$ the numbers of points, $C$ the number of input channels, $Cl$ the number of action class labels, and $B$ is the batch size.
• Figure : Figure 4: The sub-manifold multi-scale temporal convolutional layer (MS-TCN), consisting of purely temporal convolutions acting on 4D voxelized space at various kernel sizes, with input voxel resolution $(T', G'_x, G'_y, G'_z)$.
• Figure : Figure 5: The 4D Sub-Manifold Residual Bottleneck Layers. Note that kernel sizes, stride, and padding values are broadcast to 4 dimensions.