Cross-Block Fine-Grained Semantic Cascade for Skeleton-Based Sports Action Recognition

TL;DR

The paper tackles the challenge of fine-grained temporal discrimination in skeleton-based sports action recognition by introducing CFSC, a cross-block semantic cascade that aggregates low-level features from multiple GCN blocks through short-term temporal convolutions. By fusing multi-granularity information and leveraging a lightweight temporal pathway, CFSC enhances discriminative cues for fast, subtle sports actions. Empirical results on FD-7 and FSD-10 show consistent performance gains across backbones, confirming the value of cross-block feature fusion for fine-grained sports activity analysis. The work also provides a new fencing dataset (FD-7) and thorough ablations to guide future refinement of block selection and temporal-cascade design.

Abstract

Human action video recognition has recently attracted more attention in applications such as video security and sports posture correction. Popular solutions, including graph convolutional networks (GCNs) that model the human skeleton as a spatiotemporal graph, have proven very effective. GCNs-based methods with stacked blocks usually utilize top-layer semantics for classification/annotation purposes. Although the global features learned through the procedure are suitable for the general classification, they have difficulty capturing fine-grained action change across adjacent frames -- decisive factors in sports actions. In this paper, we propose a novel ``Cross-block Fine-grained Semantic Cascade (CFSC)'' module to overcome this challenge. In summary, the proposed CFSC progressively integrates shallow visual knowledge into high-level blocks to allow networks to focus on action details. In particular, the CFSC module utilizes the GCN feature maps produced at different levels, as well as aggregated features from proceeding levels to consolidate fine-grained features. In addition, a dedicated temporal convolution is applied at each level to learn short-term temporal features, which will be carried over from shallow to deep layers to maximize the leverage of low-level details. This cross-block feature aggregation methodology, capable of mitigating the loss of fine-grained information, has resulted in improved performance. Last, FD-7, a new action recognition dataset for fencing sports, was collected and will be made publicly available. Experimental results and empirical analysis on public benchmarks (FSD-10) and self-collected (FD-7) demonstrate the advantage of our CFSC module on learning discriminative patterns for action classification over others.

Figures (8)

• Figure 1: Examples of Skeleton sequences from our self-built fencing dataset FD-7. (a) step forward, (b) two-step forward, (c) step backward, (d) thrust in place, (e) lunge in place, (f) step forward lunge, and (g) sprint.
• Figure 2: Partial schematic diagram of the network in a041 where potential fine-grained information is captured by aggregating feature maps from different depth blocks.
• Figure 3: Overview of CFSC Module. $M$ blocks of varying depths are selected from the GCN backbone to obtain a collection of feature maps that encompass different granularities of information: $\left\{\mathbf{f}_{1}, \mathbf{f}_{v},...,\mathbf{f}_{M}\right \}$. At each level, $\mathbf{f}_{v}$ is integrated with $\mathbf{F}_{v-1}$ obtained first from the preceding level. Next, temporal convolution is used to aggregate features of different granularities along the temporal dimension. The resulting feature $\mathbf{F}_{v}$ is then filtered by $\lambda$ and passed to the subsequent level. The aggregated highest-level feature $\mathbf{F}_{M}$, undergoes NL and ReLU, yielding the auxiliary discriminative feature $\mathbf{F}_{dis}$.
• Figure 4: Architecture overview. The HD-GCN backbone network, contained within the blue box, accepts skeletal sequences as input. The auxiliary feature $\mathbf{F}_{std}$ generated by the CFSC Module is aggregated with $\mathbf{f}_{10}$ and subsequently passed into the FC layer for generating prediction labels.
• Figure 5: Several examples from the FD-7 dataset are displayed in the figure. Action classification in the same sport involves precise categorization with slight differences between classes. For fencing footwork $\left (a\right )$, $\left (b\right )$, $\left (c\right )$, the fencer maintains a standard stance with the right foot forward and toes pointing straight ahead. During offensive actions $\left (e\right )$, $\left (f\right )$, $\left (g\right )$, the fencer leans forward with body weight shifted and executes an upward strike using the sword hand. The extraction and use of fine-grained information within the network is recommended to enhance sports action classification accuracy.