Multimodal Fusion of Skeleton Dynamics and Clinical Gait Features for Video-Based Cerebral Palsy Severity Assessment

Abstract

Video-based gait analysis has become a promising approach for assessing motor impairment in children with cerebral palsy (CP). However, existing methods usually rely on either pose sequences or handcrafted gait features alone, making it difficult to simultaneously capture spatiotemporal motion patterns and clinically meaningful biomechanical information. To address this gap, we propose a multimodal fusion framework that integrates skeleton dynamics with contribution-guided clinically meaningful gait features. First, Grad-CAM analysis on a pre-trained ST-GCN backbone identified the most discriminative body keypoints, providing an interpretable basis for subsequent gait feature extraction. We then build a dual-stream architecture, with one stream modeling skeleton dynamics using ST-GCN and the other encoding gait geatures derived from the identified keypoints. By fusing the two streams through feature cross-attention improved four-level CP motor severity classification to 70.86%, outperforming the baseline by 5.6 percentage points. Overall, this work suggests that integrating skeleton dynamics with clinically meaningful gait descriptors can improve both prediction performance and biomechanical interpretability for video-based CP severity assessment.

Figures (1)

• Figure 1: Overview of the proposed two-step framework. Step 1: An ST-GCN backbone is pre-trained and analyzed via Grad-CAM to identify discriminative keypoints, guiding gait feature selection. Step 2: A dual-stream architecture combines skeleton dynamics (ST-GCN backbone $\to$ global average pooling $\to$$\mathbf{f}_s \in \mathbb{R}^{256}$) with extracted clinical gait features (MLP projection $\to$$\hat{\mathbf{f}}_c \in \mathbb{R}^{256}$) via feature fusion for GMFCS classification.