AnxietyFaceTrack: A Smartphone-Based Non-Intrusive Approach for Detecting Social Anxiety Using Facial Features
Nilesh Kumar Sahu, Snehil Gupta, Haroon R Lone
TL;DR
Social Anxiety Disorder lacks objective biomarkers, hindering early detection. AnxietyFaceTrack presents a smartphone-based, non-intrusive approach that analyzes facial cues during naturalistic social interactions to classify anxiety states. Using 669 OpenFace facial features from 2-minute interaction videos of 85–91 participants and a Random Forest classifier, the study reports multiclass accuracy around 91% and binary accuracies around 92–93%, with 3D landmarks and head pose driving performance. SHAP analysis provides interpretability, revealing informative facial cues, while bias analyses indicate gender- and education-related differences that warrant fairness considerations. Overall, the work demonstrates the feasibility of continuous, low-cost anxiety monitoring with potential real-world impact, subject to validation in uncontrolled settings and further bias mitigation.
Abstract
Social Anxiety Disorder (SAD) is a widespread mental health condition, yet its lack of objective markers hinders timely detection and intervention. While previous research has focused on behavioral and non-verbal markers of SAD in structured activities (e.g., speeches or interviews), these settings fail to replicate real-world, unstructured social interactions fully. Identifying non-verbal markers in naturalistic, unstaged environments is essential for developing ubiquitous and non-intrusive monitoring solutions. To address this gap, we present AnxietyFaceTrack, a study leveraging facial video analysis to detect anxiety in unstaged social settings. A cohort of 91 participants engaged in a social setting with unfamiliar individuals and their facial videos were recorded using a low-cost smartphone camera. We examined facial features, including eye movements, head position, facial landmarks, and facial action units, and used self-reported survey data to establish ground truth for multiclass (anxious, neutral, non-anxious) and binary (e.g., anxious vs. neutral) classifications. Our results demonstrate that a Random Forest classifier trained on the top 20% of features achieved the highest accuracy of 91.0% for multiclass classification and an average accuracy of 92.33% across binary classifications. Notably, head position and facial landmarks yielded the best performance for individual facial regions, achieving 85.0% and 88.0% accuracy, respectively, in multiclass classification, and 89.66% and 91.0% accuracy, respectively, across binary classifications. This study introduces a non-intrusive, cost-effective solution that can be seamlessly integrated into everyday smartphones for continuous anxiety monitoring, offering a promising pathway for early detection and intervention.