Foundation Model-based Evaluation of Neuropsychiatric Disorders: A Lifespan-Inclusive, Multi-Modal, and Multi-Lingual Study

TL;DR

FEND presents a lifespan-inclusive, multi-modal benchmark for neuropsychiatric disorder detection using acoustic and linguistic foundation models across AD, depression, and ASD. It systematically evaluates mono- and multi-modal approaches on 13 multilingual datasets, revealing modality- and language-dependent strengths, while uncovering challenges in cross-lingual generalization and modality imbalance. The study demonstrates that multi-modal fusion benefits AD and depression but struggles with ASD due to data heterogeneity, and it emphasizes the need for domain-specific fusion strategies and robust generalization methods. By providing standardized evaluation protocols and extensive benchmarks, FEND aims to guide fair comparisons, reproducible research, and practical deployment in diverse clinical settings.

Abstract

Neuropsychiatric disorders, such as Alzheimer's disease (AD), depression, and autism spectrum disorder (ASD), are characterized by linguistic and acoustic abnormalities, offering potential biomarkers for early detection. Despite the promise of multi-modal approaches, challenges like multi-lingual generalization and the absence of a unified evaluation framework persist. To address these gaps, we propose FEND (Foundation model-based Evaluation of Neuropsychiatric Disorders), a comprehensive multi-modal framework integrating speech and text modalities for detecting AD, depression, and ASD across the lifespan. Leveraging 13 multi-lingual datasets spanning English, Chinese, Greek, French, and Dutch, we systematically evaluate multi-modal fusion performance. Our results show that multi-modal fusion excels in AD and depression detection but underperforms in ASD due to dataset heterogeneity. We also identify modality imbalance as a prevalent issue, where multi-modal fusion fails to surpass the best mono-modal models. Cross-corpus experiments reveal robust performance in task- and language-consistent scenarios but noticeable degradation in multi-lingual and task-heterogeneous settings. By providing extensive benchmarks and a detailed analysis of performance-influencing factors, FEND advances the field of automated, lifespan-inclusive, and multi-lingual neuropsychiatric disorder assessment. We encourage researchers to adopt the FEND framework for fair comparisons and reproducible research.

Figures (3)

• Figure 1: Overview of the FEND Framework for Neuropsychiatric Disorder Detection. The FEND framework includes two main processes: Mono-modal and multi-modal detection. In the mono-modal process, speech or text is fed into a foundation model to extract representations, followed by a disease classification using an MLP. In the mono-modal process, speech and text are separately processed by foundation models to extract representations, which are then fused using classical multi-modal methods for a final disease prediction.
• Figure 2: Performance comparison of multi-modal fusion with and without modality imbalance mitigation techniques (OGM-GE and PMR) across different datasets. Mono-modal: The best mono-modal performance. Multi-modal: The best multi-modal performance.
• Figure 3: Cross-corpus inference results. (a) and (b) present the cross-corpus inference results for AD and depression, respectively. The x-axis denotes the test datasets. Legends represent the WF1 scores of models pre-trained on different datasets when evaluated on the test datasets. "IC" denotes the "Intra-Corpus" experimental results. The "---" indicates the same performance obtained on the two marked datasets.