Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognition

Jinfu Liu, Baiqiao Yin, Jiaying Lin, Jiajun Wen, Yue Li, Mengyuan Liu

TL;DR

The paper addresses skeleton-based action recognition by moving beyond single-backbone models and proposes the Hybrid Dual-Branch Network (HDBN), which fuses a Graph Convolutional Network (MixGCN) and a Transformer-based backbone (MixFormer) to jointly exploit local graph structure and global information. It processes multiple skeleton modalities (J, B, JM, BM) and leverages a 3D pose estimator to derive 3D data from 2D inputs, enabling robust 2D/3D fusion. Key contributions include the use of three dynamic adjacency GCNs, the Skeleton MixFormer transformer backbone, and a late-fusion ensemble that delivers strong performance on UAV-Human CSv1/CSv2 benchmarks. On UAV-Human, HDBN achieves 47.95% and 75.36% top-1 accuracy on CSv1 and CSv2, respectively, outperforming many existing methods and highlighting the practical benefit of backbone complementarity for skeleton-only action recognition.

Abstract

Skeleton-based action recognition has gained considerable traction thanks to its utilization of succinct and robust skeletal representations. Nonetheless, current methodologies often lean towards utilizing a solitary backbone to model skeleton modality, which can be limited by inherent flaws in the network backbone. To address this and fully leverage the complementary characteristics of various network architectures, we propose a novel Hybrid Dual-Branch Network (HDBN) for robust skeleton-based action recognition, which benefits from the graph convolutional network's proficiency in handling graph-structured data and the powerful modeling capabilities of Transformers for global information. In detail, our proposed HDBN is divided into two trunk branches: MixGCN and MixFormer. The two branches utilize GCNs and Transformers to model both 2D and 3D skeletal modalities respectively. Our proposed HDBN emerged as one of the top solutions in the Multi-Modal Video Reasoning and Analyzing Competition (MMVRAC) of 2024 ICME Grand Challenge, achieving accuracies of 47.95% and 75.36% on two benchmarks of the UAV-Human dataset by outperforming most existing methods. Our code will be publicly available at: https://github.com/liujf69/ICMEW2024-Track10.

HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognition

TL;DR

The paper addresses skeleton-based action recognition by moving beyond single-backbone models and proposes the Hybrid Dual-Branch Network (HDBN), which fuses a Graph Convolutional Network (MixGCN) and a Transformer-based backbone (MixFormer) to jointly exploit local graph structure and global information. It processes multiple skeleton modalities (J, B, JM, BM) and leverages a 3D pose estimator to derive 3D data from 2D inputs, enabling robust 2D/3D fusion. Key contributions include the use of three dynamic adjacency GCNs, the Skeleton MixFormer transformer backbone, and a late-fusion ensemble that delivers strong performance on UAV-Human CSv1/CSv2 benchmarks. On UAV-Human, HDBN achieves 47.95% and 75.36% top-1 accuracy on CSv1 and CSv2, respectively, outperforming many existing methods and highlighting the practical benefit of backbone complementarity for skeleton-only action recognition.

Abstract

Skeleton-based action recognition has gained considerable traction thanks to its utilization of succinct and robust skeletal representations. Nonetheless, current methodologies often lean towards utilizing a solitary backbone to model skeleton modality, which can be limited by inherent flaws in the network backbone. To address this and fully leverage the complementary characteristics of various network architectures, we propose a novel Hybrid Dual-Branch Network (HDBN) for robust skeleton-based action recognition, which benefits from the graph convolutional network's proficiency in handling graph-structured data and the powerful modeling capabilities of Transformers for global information. In detail, our proposed HDBN is divided into two trunk branches: MixGCN and MixFormer. The two branches utilize GCNs and Transformers to model both 2D and 3D skeletal modalities respectively. Our proposed HDBN emerged as one of the top solutions in the Multi-Modal Video Reasoning and Analyzing Competition (MMVRAC) of 2024 ICME Grand Challenge, achieving accuracies of 47.95% and 75.36% on two benchmarks of the UAV-Human dataset by outperforming most existing methods. Our code will be publicly available at: https://github.com/liujf69/ICMEW2024-Track10.
Paper Structure (17 sections, 5 equations, 2 figures, 6 tables)

This paper contains 17 sections, 5 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CNN-Based and RNN-Based Methods
  4. GCN-Based Methods
  5. Transformer-Based Methods
  6. Hybrid Dual-Branch Methods
  7. Method
  8. Multimodal Data Processing
  9. MixGCN Backbone
  10. MixFormer Backbone
  11. Ensemble
  12. Experiments
  13. Datasets and Implementation Details
  14. Comparisons with the State-of-the-Art
  15. Ablation Studies
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Framework of our proposed Hybrid Dual-branch Network.
  • Figure 2: The confusion matrix of UAV-Human. The more yellow the squares on the diagonal, the more accurate the recognition. (a) UAV-Human dataset on the benchmark of CSv1. (b) UAV-Human dataset on the benchmark of CSv2