VoxCog: Towards End-to-End Multilingual Cognitive Impairment Classification through Dialectal Knowledge

TL;DR

VoxCog proposes an end-to-end, dialect-aware framework for cognitive impairment classification from raw speech by transferring dialect representations learned from Voxlect. The model initializes with pretrained dialect weights and fine-tunes using LoRA on 15-second speech segments, enabling multilingual, text-free detection of AD and MCI. Across six datasets in English and non-English, VoxCog achieves state-of-the-art or competitive results, outperforming many multimodal and language-model baselines while avoiding ASR and additional modalities. The work highlights dialectal phonetic and prosodic patterns as a robust prior for identifying atypical speech due to cognitive decline, with a scalable, language-inclusive screening potential.

Abstract

In this work, we present a novel perspective on cognitive impairment classification from speech by integrating speech foundation models that explicitly recognize speech dialects. Our motivation is based on the observation that individuals with Alzheimer's Disease (AD) or mild cognitive impairment (MCI) often produce measurable speech characteristics, such as slower articulation rate and lengthened sounds, in a manner similar to dialectal phonetic variations seen in speech. Building on this idea, we introduce VoxCog, an end-to-end framework that uses pre-trained dialect models to detect AD or MCI without relying on additional modalities such as text or images. Through experiments on multiple multilingual datasets for AD and MCI detection, we demonstrate that model initialization with a dialect classifier on top of speech foundation models consistently improves the predictive performance of AD or MCI. Our trained models yield similar or often better performance compared to previous approaches that ensembled several computational methods using different signal modalities. Particularly, our end-to-end speech-based model achieves 87.5% and 85.9% accuracy on the ADReSS 2020 challenge and ADReSSo 2021 challenge test sets, outperforming existing solutions that use multimodal ensemble-based computation or LLMs.

Figures (2)

• Figure 1: Overview of the VoxCog framework. We initialize the VoxCog model training with the model weights from predicting the corresponding dialects. The figure uses sources from the OpenMoji web.
• Figure 2: Overview of the data processing in VoxCog.