Deep Learning Pose Estimation for Multi-Label Recognition of Combined Hyperkinetic Movement Disorders

TL;DR

The work tackles the challenge of distinguishing overlapping hyperkinetic movement disorders (HMDs) from routine clinical videos by proposing a pose-based deep-learning pipeline that converts 2D keypoints into anatomically meaningful time-series features. It integrates window-level screening with 10-second segments, subject-level multi-label inference using percentile pooling ($p_{90}$) of window probabilities, and thresholding guided by clinical constraints ($\tau_{\mathcal{l}}$), achieving robust discrimination across eight HMD phenotypes (e.g., dystonia, tremor, chorea) with strong subject-level performance. The approach emphasizes interpretability by mapping decision-level importance to clinically grounded kinematic families and anatomical regions, revealing predominant contributions from baseline posture and excursions, especially in cranial and proximal upper-limb landmarks. The findings support the feasibility of scalable, objective, and explainable video-based phenotyping for combined HMDs, with clear avenues for external validation, uncertainty-aware labeling, and prospective integration into screening, monitoring, and therapeutic evaluation workflows.

Abstract

Hyperkinetic movement disorders (HMDs) such as dystonia, tremor, chorea, myoclonus, and tics are disabling motor manifestations across childhood and adulthood. Their fluctuating, intermittent, and frequently co-occurring expressions hinder clinical recognition and longitudinal monitoring, which remain largely subjective and vulnerable to inter-rater variability. Objective and scalable methods to distinguish overlapping HMD phenotypes from routine clinical videos are still lacking. Here, we developed a pose-based machine-learning framework that converts standard outpatient videos into anatomically meaningful keypoint time series and computes kinematic descriptors spanning statistical, temporal, spectral, and higher-order irregularity-complexity features.

Figures (6)

• Figure 1: Schematic overview of the video-to-classification pipeline for automated detection of hyperkinetic movement disorders (HMDs). Clinical videos are decoded frame-by-frame using OpenCV, and 2D human pose estimation is performed with YOLOv8 to detect 17 anatomical keypoints per frame. Keypoint coordinates are converted to pixel space and used to construct per-frame signals, including landmark displacement signals. Frames are indexed into consecutive 10-second windows and stored using pandas. For each window, statistical, temporal, spectral and complexity descriptors are computed from each displacement signal, yielding a window-level feature vector. Clinical expert annotations are aligned with the extracted features at the 10-second window level, with uncertain segments excluded. From this merged window-level dataset, two analysis branches are derived: (i) a binary screening dataset contrasting symptom-positive patient windows against control windows, and (ii) an 8-label multi-label dataset where each window is represented by a symptom-presence vector (including controls and all-zero windows when enabled). In the multi-label branch, one model is trained per symptom label at the window level; window-level probabilities are then aggregated at the subject level (e.g., p90 pooling) and label-specific decision thresholds are selected using training data under constraints on control false positives/specificity. Performance is evaluated using ROC-AUC/AUPRC and multi-label metrics (e.g., Hamming and Jaccard). Example images and schematic outputs are shown using data from a healthy control who provided informed consent for publication.
• Figure 2: Window-based symptom classification performance and condition-specific comparison across rest, posture maintain, and with action. (A) Summary of the window-based binary classification of symptom presence versus absence for each hyperkinetic movement disorder. For each symptom, the table reports the mean F1-score for the positive class (averaged across available raters and shown with rater-specific values), the percentage of patients correctly identified at the patient level (derived from window-level predictions), and an inter-rater consistency indicator ($\Delta$F1 between raters; lower values indicate higher agreement). Cell background colors provide a qualitative performance cue (green = higher performance; orange = intermediate; red = lower; for $\Delta$F1, green = lower disagreement). (B) Performance of symptom classification pipelines evaluated on condition-specific 10-second segments, stratified by motor context (Rest, Action, Posture). For each symptom--condition pair, bars show the mean F1-score (positive class) $\pm$ standard deviation across cross-validation folds for a single representative model per classifier family (XGBoost, LightGBM, RandomForest, Logistic Regression, SVM, k-NN, MLP), enabling direct comparison of model families and motor conditions.
• Figure 3: Patient-level multi-label performance for simultaneous detection of hyperkinetic movement disorders (HMDs) using p90 aggregation of window-level probabilities. (A) Confusion-matrix decomposition for the best-performing per-label pipelines, showing the number of subjects classified as true negatives (TN), true positives (TP), false positives (FP) and false negatives (FN) for each phenotype. Counts are reported at the subject level (total subjects per label = TN+TP+FP+FN) to highlight the relative contribution of correct detections and error types across phenotypes, with performance interpreted in the context of label prevalence. (B) Comparison of the best overall pipelines selected by different evaluation criteria (macro-AUPRC, macro-AUC, sample-wise Jaccard index, Hamming accuracy, and 1 - clinical cost), reporting mean $\pm$ s.d. across cross-validation folds at the subject level. This panel illustrates that different operating objectives select different model configurations and summarizes the trade-offs between discrimination (AUC/AUPRC) and error-sensitive objectives (Jaccard/Hamming/clinical cost) under the same p90 subject-level aggregation and per-label thresholding policy.
• Figure 4: Patient-level performance for dystonia, chorea, and athetosis. For each phenotype, we report the confusion matrix (TP, FN, FP, TN) at the patient level, using the single best-performing model for that label as defined by the lowest patient-level error count (FP+FN). The top row corresponds to patients with the phenotype present in the ground truth ("Actual 1"), and the bottom row to phenotype absent ("Actual 0"); columns indicate model predictions ("Pred 1" vs "Pred 0"). In addition to error counts, each panel summarizes key discrimination and classification metrics computed from cross-validation (AUC, AUPRC, precision, recall, specificity, and F1). Dystonia shows near-perfect detection (no false negatives) with a single false positive, while chorea and athetosis exhibit higher residual error rates driven by both false positives and false negatives.
• Figure 5: Clinically interpretable decision-level feature importance at the patient level. (A) Decision-level feature importance by kinematic family. Patient-level permutation importance was computed exclusively on the held-out outer test folds (n_folds $\geq$ 2 stability criterion) and summarized for the best-by-Hamming model (MinMaxScaler + SVM, C=3.0, $\gamma$=0.03). Importance was defined as the increase in patient-level decision error induced by permuting one feature at a time, expressed as $\Delta$ error where error = (FP+FN)/N patients, while preserving the full inference pathway (window-level probabilities, p90 patient aggregation, and label-specific clinical thresholding). Bars report the normalized share of $\Delta$ error attributed to each clinically grounded kinematic family (baseline posture, sustained bias, excursions, variability, rhythmicity, directionality, and irregularity/complexity), with percentages indicating each family's contribution within symptom. Across phenotypes, importance concentrates in postural set-point descriptors (baseline posture and sustained bias) and extreme excursions, whereas rhythmicity and irregularity/complexity contribute more selectively, consistent with phenotype-specific motor phenomenology. (B) Anatomical distribution of decision-level importance. The same stable, decision-level attributions were aggregated by landmark region (head/face, upper limb, lower limb) and displayed as the normalized share of $\Delta$ error per symptom (color intensity and in-panel percentages). Head/face and proximal upper-limb descriptors account for the majority of decision-level evidence for several phenotypes, consistent with robust visibility and tracking of axial/proximal landmarks in standard clinical videos. Lower-limb contributions are more phenotype-dependent, increasing for selected phenotypes, consistent with task- and field-of-view--dependent expression of discriminative movement patterns.