BHaRNet: Reliability-Aware Body-Hand Modality Expertized Networks for Fine-grained Skeleton Action Recognition

TL;DR

BHaRNet tackles the challenge of fine-grained skeleton action recognition by mitigating reliability asymmetry between body and hand cues. It introduces calibration-free skeleton learning, a Noisy-OR reliability-aware fusion, and a unified intra- to cross-modal ensemble that extends skeletal cues to RGB, enabling robust skeleton–RGB action recognition. The approach delivers state-of-the-art or competitive results on NTU RGB+D 60/120 and PKU-MMD, with strong hand-centric performance on NTU-Hand 11/27, while maintaining efficiency. The framework demonstrates improved robustness to occlusion, noise, and viewpoint changes, and provides a practical path toward reliable multi-modal HAR in real-world settings.

Abstract

Skeleton-based human action recognition (HAR) has achieved remarkable progress with graph-based architectures. However, most existing methods remain body-centric, focusing on large-scale motions while neglecting subtle hand articulations that are crucial for fine-grained recognition. This work presents a probabilistic dual-stream framework that unifies reliability modeling and multi-modal integration, generalizing expertized learning under uncertainty across both intra-skeleton and cross-modal domains. The framework comprises three key components: (1) a calibration-free preprocessing pipeline that removes canonical-space transformations and learns directly from native coordinates; (2) a probabilistic Noisy-OR fusion that stabilizes reliability-aware dual-stream learning without requiring explicit confidence supervision; and (3) an intra- to cross-modal ensemble that couples four skeleton modalities (Joint, Bone, Joint Motion, and Bone Motion) to RGB representations, bridging structural and visual motion cues in a unified cross-modal formulation. Comprehensive evaluations across multiple benchmarks (NTU RGB+D~60/120, PKU-MMD, N-UCLA) and a newly defined hand-centric benchmark exhibit consistent improvements and robustness under noisy and heterogeneous conditions.

Figures (8)

• Figure 1: Visualization of two hand-centric actions—"Yawn”(left) and "Hush”(right)—cropped frames from NTU RGB+D with body skeleton. Both representations share nearly identical body postures, indicating strong global pose similarity across distinct hand gestures. This highlights the challenge of distinguishing fine-grained actions using only body skeletons and motivates the need for reliability-aware hand modeling.
• Figure 2: Overview of the dual-stream architectures. Left: BHaRNet-P with interactive body (BI) and hand (HI) branches connected via lightweight cross-attention. Right: BHaRNet-E with additional expertized branches (BE, HE) that preserve modality-specific cues while sharing context through the interactive branches. The corresponding branch–loss configurations for deterministic baseline and probabilistic framework are summarized in Table \ref{['tab:branch-loss-map']}.
• Figure 3: Motivating example for the calibration-free representation learning used in our generalized framework. We visualize two consecutive frames at 30 fps from a single sequence. Left: frame 17 with a accurate hand estimate. Right: frame 18 where the index and middle fingers are corrupted by noise. Rows show (top) RGB crops with hand estimation, (middle) native 3D hand skeletons, and (bottom) 3D hand skeletons after canonical-space transformation. All 3D views share the same viewpoint and axis scales for fair comparison. In the native space, the overall hand configuration remains stable except around the noisy index and middle joints. After canonical-space transformation, local noise propagates to the entire hand, causing large joint-wise displacements and noticeable shape distortion.
• Figure 4: Schematic of our BHaRNet-M. We integrate BHaRNet-E for skeletal streams and add an RGB stream with its own training path (bold lines). The RGB branch receives body-joint guidance from the body-expertized branch, focusing the visual feature extractor on relevant spatio-temporal regions.
• Figure 5: Accuracy–GFLOPs trade-off for skeleton-based action recognition on NTU 120 cross-subject. Red-toned markers denote our probabilistic models (BHaRNet-B/E/P), and green-to-blue markers denote previous skeleton-based state-of-the-art methods (DeGCN, ProtoGCN, 3MFormer, SkeleT).