Unmasking Parkinson's Disease with Smile: An AI-enabled Screening Framework

TL;DR

Smiling videos can effectively differentiate between individuals with and without PD, offering a potentially easy, accessible, and cost-efficient way to screen for PD, especially when a clinical diagnosis is difficult to access.

Abstract

We present an efficient and accessible PD screening method by leveraging AI-driven models enabled by the largest video dataset of facial expressions from 1,059 unique participants. This dataset includes 256 individuals with PD, 165 clinically diagnosed, and 91 self-reported. Participants used webcams to record themselves mimicking three facial expressions (smile, disgust, and surprise) from diverse sources encompassing their homes across multiple countries, a US clinic, and a PD wellness center in the US. Facial landmarks are automatically tracked from the recordings to extract features related to hypomimia, a prominent PD symptom characterized by reduced facial expressions. Machine learning algorithms are trained on these features to distinguish between individuals with and without PD. The model was tested for generalizability on external (unseen during training) test videos collected from a US clinic and Bangladesh. An ensemble of machine learning models trained on smile videos achieved an accuracy of 87.9+-0.1% (95% Confidence Interval) with an AUROC of 89.3+-0.3% as evaluated on held-out data (using k-fold cross-validation). In external test settings, the ensemble model achieved 79.8+-0.6% accuracy with 81.9+-0.3% AUROC on the clinical test set and 84.9+-0.4% accuracy with 81.2+-0.6% AUROC on participants from Bangladesh. In every setting, the model was free from detectable bias across sex and ethnic subgroups, except in the cohorts from Bangladesh, where the model performed significantly better for female participants than males. Smiling videos can effectively differentiate between individuals with and without PD, offering a potentially easy, accessible, and cost-efficient way to screen for PD, especially when a clinical diagnosis is difficult to access.

Figures (6)

• Figure 1: A brief overview of the AI-based system for classifying individuals with and without PD. Anyone can record their smile expressions in front of a computer webcam. The system extracts facial keypoints for each frame using Openface and Mediapipe tools, and then computes features by summarizing the temporal dimensions with statistical aggregates. Top-$n$ features are then fed to an ensemble of models which finally decides whether the participant has PD or not.
• Figure 2: Feature visualization and model performance for each facial expression. All the extracted features from training videos ($n = 827$) of (a) smile, (b) disgust, and (c) surprise expression are reduced into two principal components for visualization. The orange and blue dots separate participants with and without PD. The ROC curve for the predictive model trained on features extracted only from the (d) smile, (e) disgust, and (f) surprise expression and evaluated on held-out data demonstrates how well the model can separate participants with and without PD using features from a single expression.
• Figure 3: PD screening from videos of smiling. ROC curves for differentiating between participants with and without PD (a) on held-out data with k-fold cross validation ($n = 827$), (b) on external test data collected at a U.S. clinic ($n = 75$), and (c) on external home-recorded test data collected from Bangladesh ($n = 149$). The models only used features from the smile expression. The corresponding confusion matrices are displayed below the ROC curves (d-f).
• Figure 4: Comprehensive analysis of model performance across subgroups and PD durations. The first three plots demonstrate the rate of (a) miss-classification, (b) underdiagnosis, and (c) overdiagnosis across population subgroups based on sex, ethnicity, and age. The last plot visualizes the (d) miss-classification rate across different bins of disease duration. Durations are binned into yearly intervals, with sample sizes for each interval noted in parentheses. In all cases, The error bars represent $95\%$ confidence interval.
• Figure 5: SHAP beeswarm plot showing the impact of features on the model's output. The plot displays the top features contributing to the model, with the color representing the feature value (red for high, blue for low). The "Sum of 15 other features" combines the contributions of less significant features.