CogniVoice: Multimodal and Multilingual Fusion Networks for Mild Cognitive Impairment Assessment from Spontaneous Speech

TL;DR

CogniVoice tackles MCI detection and MMSE estimation from spontaneous speech in a multilingual setting by integrating multimodal signals through a Product of Experts ensemble. It combines Whisper-based speech features, language-aware text features from BERT, and DisVoice acoustic cues, jointly optimizing MCI classification and MMSE regression. By mitigating shortcut learning and cross-language mismatch, CogniVoice achieves substantial gains over strong baselines (e.g., F1 and RMSE improvements) and reduces language-related performance gaps, highlighting its practical potential for scalable, language-robust cognitive health screening from speech.

Abstract

Mild Cognitive Impairment (MCI) is a medical condition characterized by noticeable declines in memory and cognitive abilities, potentially affecting individual's daily activities. In this paper, we introduce CogniVoice, a novel multilingual and multimodal framework to detect MCI and estimate Mini-Mental State Examination (MMSE) scores by analyzing speech data and its textual transcriptions. The key component of CogniVoice is an ensemble multimodal and multilingual network based on ``Product of Experts'' that mitigates reliance on shortcut solutions. Using a comprehensive dataset containing both English and Chinese languages from TAUKADIAL challenge, CogniVoice outperforms the best performing baseline model on MCI classification and MMSE regression tasks by 2.8 and 4.1 points in F1 and RMSE respectively, and can effectively reduce the performance gap across different language groups by 0.7 points in F1.

Figures (2)

• Figure 1: Schematic diagram of the proposed approach. (a) Overall architecture of CogniVoice: a multi-feature model is fused with different uni-feature models through product-of-experts (PoE) to mitigate overfitting and prioritize true task signals over spurious features. (b) mitigating overfitting with PoE. (c) Example of speech provided by a patient diagnosed with MCI describing the cookie theft picture borod1980normative for MCI assessment. The description lacks coherence and clear transitions between events, lack to remember or identify the objects and use vague language highlighted in the text, which are important signs of cognitive impairment. Non-MCI patients often narrate a sequence of events with clarity, identify various elements in the picture, provide detailed description, recall objects and events already described.
• Figure 2: Ablation study on multimodal features.