FineTec: Fine-Grained Action Recognition Under Temporal Corruption via Skeleton Decomposition and Sequence Completion

TL;DR

FineTec addresses fine-grained action recognition under temporal corruption by integrating context-aware sequence completion, biologically informed spatial decomposition, and physics-driven acceleration modeling. The framework restores missing frames, enhances subtle motion cues by dynamic/static region processing, and uses Lagrangian dynamics to obtain discriminative acceleration features fused with pose sequences for GCN-based recognition. It introduces Gym288-skeleton to benchmark fine-grained skeleton actions and demonstrates state-of-the-art performance on Gym99/Gym288 and NTU datasets across varying corruption levels, validating robustness and generalization. This approach advances practical action understanding in real-world scenarios where online pose estimation yields substantial data loss and irregular temporal sampling.

Abstract

Recognizing fine-grained actions from temporally corrupted skeleton sequences remains a significant challenge, particularly in real-world scenarios where online pose estimation often yields substantial missing data. Existing methods often struggle to accurately recover temporal dynamics and fine-grained spatial structures, resulting in the loss of subtle motion cues crucial for distinguishing similar actions. To address this, we propose FineTec, a unified framework for Fine-grained action recognition under Temporal Corruption. FineTec first restores a base skeleton sequence from corrupted input using context-aware completion with diverse temporal masking. Next, a skeleton-based spatial decomposition module partitions the skeleton into five semantic regions, further divides them into dynamic and static subgroups based on motion variance, and generates two augmented skeleton sequences via targeted perturbation. These, along with the base sequence, are then processed by a physics-driven estimation module, which utilizes Lagrangian dynamics to estimate joint accelerations. Finally, both the fused skeleton position sequence and the fused acceleration sequence are jointly fed into a GCN-based action recognition head. Extensive experiments on both coarse-grained (NTU-60, NTU-120) and fine-grained (Gym99, Gym288) benchmarks show that FineTec significantly outperforms previous methods under various levels of temporal corruption. Specifically, FineTec achieves top-1 accuracies of 89.1% and 78.1% on the challenging Gym99-severe and Gym288-severe settings, respectively, demonstrating its robustness and generalizability. Code and datasets could be found at https://smartdianlab.github.io/projects-FineTec/.

Figures (11)

• Figure 1: (a) Illustration of the challenging task: Fine-Grained Action Recognition under Temporal Corruption. (b) Compared to other GCN-based methods, the proposed FineTec framework can restore corrupted skeleton sequences and extract more discriminative features for recognition through context-aware completion, skeleton-based decomposition, and physics-driven modeling.
• Figure 2: Overview of the Pipeline. FineTec consists of three core modules: ① Context-aware Sequence Completion restores missing or corrupted skeleton frames using in-context learning, producing $S_{base}$; ② Skeleton-based Spatial Decomposition partitions $S_{base}$ into anatomical regions by motion intensity, generating dynamic ($S_{dyna}$) and static ($S_{stat}$) variants, which are fused into $S_{pred}$; ③ Physics-driven Acceleration Modeling infers joint accelerations via Lagrangian dynamics and data-driven finite differences, producing fused temporal dynamics features $\mathbf{a}$. The resulting positional ($S_{pred}$) and dynamic (${a}_{pred}$) features are used for downstream fine-grained action recognition.
• Figure 3: The Construction Process and Statistics of the constrcucted Gym288-skeleton Dataset.
• Figure 4: Comparison of Skeleton Restoration Methods on Gym99-skeleton and Gym288-skeleton. Top-1 accuracy of FineTec (Ours), Interpolation, and Duplication is reported under varying levels of temporal corruption.
• Figure 5: Qualitative Results of Skeleton Restoration. Our context-aware completion method more accurately reconstructs missing frames and preserves fine-grained motion details compared to the ablation without in-context learning.