Explore Human Parsing Modality for Action Recognition

TL;DR

This work tackles action recognition by integrating a novel human parsing modality with skeletal data to improve robustness against appearance noise. The authors propose EPP-Net, a dual-branch architecture with a GCN-based pose branch and a CNN-based parsing branch, whose outputs are fused via late fusion $O = \alpha C(P) + \beta M(V)$. Key contributions include defining four skeleton modalities ($J$, $B$, $JM$, $BM$), generating colorized, noiseless human parsing feature maps from RGB using object detection and parsing, and leveraging a GCN update $H^{l+1} = \sigma(D^{-\frac{1}{2}} A D^{-\frac{1}{2}} H^l W^l)$. Experiments on NTU-RGB+D and NTU-RGB+D 120 demonstrate state-of-the-art performance, validating the parsing modality and the late-fusion strategy for robust action recognition.

Abstract

Multimodal-based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, we introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts, while filtering out most irrelevant noise. We propose a new dual-branch framework called Ensemble Human Parsing and Pose Network (EPP-Net), which is the first to leverage both skeletons and human parsing modalities for action recognition. The first human pose branch feeds robust skeletons in graph convolutional network to model pose features, while the second human parsing branch also leverages depictive parsing feature maps to model parsing festures via convolutional backbones. The two high-level features will be effectively combined through a late fusion strategy for better action recognition. Extensive experiments on NTU RGB+D and NTU RGB+D 120 benchmarks consistently verify the effectiveness of our proposed EPP-Net, which outperforms the existing action recognition methods. Our code is available at: https://github.com/liujf69/EPP-Net-Action.

Figures (7)

• Figure 1: Action recognition using different modalities. (a) Previous methods using unimodal modality like skeleton, RGB or point cloud data. (b) Previous multimodal methods using noisy RGB modality. (c) Our EPP-Net is the first to introduce novel human parsing modality and leverage both skeletons and human parsing feature maps for multimodal-based action recognition.
• Figure 2: Framework of our proposed Ensemble Human Parsing and Pose Network.
• Figure 3: Generate noiseless human parsing feature map from noisy RGB videos.
• Figure 4: Visualization of human parsing feature maps for different action samples. (a) Feature map of drink water. (b) Feature map of throwing. (c) Feature map of stand up. (d) Feature map of clapping. (e) Feature map of hand waving. (f) Feature map of salute.
• Figure 5: The confusion matrix of NTU60 dataset and NTU120 dataset. The more yellow the squares on the diagonal, the more accurate the recognition. (a) NTU60 dataset on the benchmark of X-Sub. (b) NTU60 dataset on the benchmark of X-View. (c) NTU120 dataset on the benchmark of X-Sub. (d) NTU120 dataset on the benchmark of X-Set.