Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parts-Mamba: Augmenting Joint Context with Part-Level Scanning for Occluded Human Skeleton

Tianyi Shen, Huijuan Xu, Nilesh Ahuja, Omesh Tickoo, Philip Shin, Vijaykrishnan Narayanan

TL;DR

This work tackles occlusion in skeleton-based action recognition by addressing the limitations of local-context GCNs. It introduces Parts-Mamba, a hybrid GCN-Mamba framework that combines Part-Wise/Body Scanning, topological graph modeling, and a gated fusion to retain distant joint context, complemented by a Mamba Temporal Encoder for temporal dynamics. The approach achieves state-of-the-art results across multiple occlusion scenarios on NTU-60 and NTU-120, including parts-specific, random, and periodic frame occlusions, with notable gains over prior methods and lower computational cost than ViT or MLPMixer baselines. Overall, Parts-Mamba enhances robustness to occlusion in real-world skeleton action recognition, offering practical improvements in accuracy and efficiency for downstream applications.

Abstract

Skeleton action recognition involves recognizing human action from human skeletons. The use of graph convolutional networks (GCNs) has driven major advances in this recognition task. In real-world scenarios, the captured skeletons are not always perfect or complete because of occlusions of parts of the human body or poor communication quality, leading to missing parts in skeletons or videos with missing frames. In the presence of such non-idealities, existing GCN models perform poorly due to missing local context. To address this limitation, we propose Parts-Mamba, a hybrid GCN-Mamba model designed to enhance the ability to capture and maintain contextual information from distant joints. The proposed Parts-Mamba model effectively captures part-specific information through its parts-specific scanning feature and preserves non-neighboring joint context via a parts-body fusion module. Our proposed model is evaluated on the NTU RGB+D 60 and NTU RGB+D 120 datasets under different occlusion settings, achieving up to 12.9% improvement in accuracy.

Parts-Mamba: Augmenting Joint Context with Part-Level Scanning for Occluded Human Skeleton

TL;DR

This work tackles occlusion in skeleton-based action recognition by addressing the limitations of local-context GCNs. It introduces Parts-Mamba, a hybrid GCN-Mamba framework that combines Part-Wise/Body Scanning, topological graph modeling, and a gated fusion to retain distant joint context, complemented by a Mamba Temporal Encoder for temporal dynamics. The approach achieves state-of-the-art results across multiple occlusion scenarios on NTU-60 and NTU-120, including parts-specific, random, and periodic frame occlusions, with notable gains over prior methods and lower computational cost than ViT or MLPMixer baselines. Overall, Parts-Mamba enhances robustness to occlusion in real-world skeleton action recognition, offering practical improvements in accuracy and efficiency for downstream applications.

Abstract

Skeleton action recognition involves recognizing human action from human skeletons. The use of graph convolutional networks (GCNs) has driven major advances in this recognition task. In real-world scenarios, the captured skeletons are not always perfect or complete because of occlusions of parts of the human body or poor communication quality, leading to missing parts in skeletons or videos with missing frames. In the presence of such non-idealities, existing GCN models perform poorly due to missing local context. To address this limitation, we propose Parts-Mamba, a hybrid GCN-Mamba model designed to enhance the ability to capture and maintain contextual information from distant joints. The proposed Parts-Mamba model effectively captures part-specific information through its parts-specific scanning feature and preserves non-neighboring joint context via a parts-body fusion module. Our proposed model is evaluated on the NTU RGB+D 60 and NTU RGB+D 120 datasets under different occlusion settings, achieving up to 12.9% improvement in accuracy.

Paper Structure

This paper contains 12 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Details of PartsMamba
  3. Part-Wise Scanning and Body Scanning
  4. Mamba with Topological Graph Modeling
  5. The Gated Mamba Fusion Module
  6. Mamba Temporal Encoder
  7. Experiments
  8. Experiment Setup
  9. Results on Parts-Specific Occlusion
  10. Results on Temporal Frame Occlusion
  11. Case Study with Other Architectures
  12. Conclusion

Figures (3)

  • Figure 1: An illustration of spatial occlusion and temporal occlusion. The figures at the top show the skeleton detected from regular video input, and the bottom figures show the occluded skeleton detected from the video input with occlusion. Note that in temporal occlusion, all the nodes are collapsed to one original node in occluded frames.
  • Figure 2: Detailed structures of the parts-body scanning module.
  • Figure 3: Performance Analysis After Replacing the PartsMamba Block by Vision Transfomer and MLPMixer.