Feasibility of Detecting Cognitive Impairment and Psychological Well-being among Older Adults Using Facial, Acoustic, Linguistic, and Cardiovascular Patterns Derived from Remote Conversations

TL;DR

The paper investigates remote, multimodal signaling of cognitive impairment and psychological well-being in older adults by extracting facial, acoustic, linguistic, and cardiovascular cues from remote conversations. It demonstrates feasibility with distinct tasks (e.g., CDR discrimination with AUROC around 0.77; LSNS 0.74; social satisfaction 0.75; psychological well-being 0.72; negative affect 0.74), highlighting speech and language patterns as key indicators of cognitive status while facial and cardiovascular signals better inform social and affective well-being. Feature-importance analyses indicate temporal speech dynamics and language cues drive cognitive assessments, whereas facial expressions and cardiovascular patterns drive well-being measures; however, significant biases by age, sex, disease state, and education persist, even when mitigation is applied. The study underscores the potential of scalable, remote monitoring for cognitive and psychological health in aging populations, while calling for larger, diverse datasets and more robust bias mitigation to ensure generalization and fairness in real-world deployment.

Abstract

The aging society urgently requires scalable methods to monitor cognitive decline and identify social and psychological factors indicative of dementia risk in older adults. Our machine learning (ML) models captured facial, acoustic, linguistic, and cardiovascular features from 39 older adults with normal cognition or Mild Cognitive Impairment (MCI), derived from remote video conversations and quantified their cognitive status, social isolation, neuroticism, and psychological well-being. Our model could distinguish Clinical Dementia Rating Scale (CDR) of 0.5 (vs. 0) with 0.77 area under the receiver operating characteristic curve (AUC), social isolation with 0.74 AUC, social satisfaction with 0.75 AUC, psychological well-being with 0.72 AUC, and negative affect with 0.74 AUC. Our feature importance analysis showed that speech and language patterns were useful for quantifying cognitive impairment, whereas facial expressions and cardiovascular patterns were useful for quantifying social and psychological well-being. Our bias analysis showed that the best-performing models for quantifying psychological well-being and cognitive states in older adults exhibited significant biases concerning their age, sex, disease condition, and education levels. Our comprehensive analysis shows the feasibility of monitoring the cognitive and psychological health of older adults, as well as the need for collecting largescale interview datasets of older adults to benefit from the latest advances in deep learning technologies to develop generalizable models across older adults with diverse demographic backgrounds and disease conditions.