Layer-Aware Early Fusion of Acoustic and Linguistic Embeddings for Cognitive Status Classification

TL;DR

This study tackles cognitive status classification from speech by examining layer-aware early fusion of acoustic and linguistic embeddings. It leverages a DementiaBank-derived dataset of 1,629 participants and evaluates multiple model pairings (wav2vec 2.0 or Whisper with DistilBERT or RoBERTa) under unimodal, early fusion, and late fusion settings, with hyperparameter tuning and repeated seeds. The key finding is that mid-encoder layers (approximately 8–10) are most informative for EF, yielding the best F1 (0.633 for EF Whisper+RoBERTa layer 9), while LF achieves the best log loss (0.678). These results show that EF enhances discrimination when acoustic representations remain acoustically driven, whereas LF offers stronger probability calibration, informing practical design choices for clinical cognitive screening pipelines.

Abstract

Speech contains both acoustic and linguistic patterns that reflect cognitive decline, and therefore models describing only one domain cannot fully capture such complexity. This study investigates how early fusion (EF) of speech and its corresponding transcription text embeddings, with attention to encoder layer depth, can improve cognitive status classification. Using a DementiaBank-derived collection of recordings (1,629 speakers; cognitively normal controls$\unicode{x2013}$CN, Mild Cognitive Impairment$\unicode{x2013}$MCI, and Alzheimer's Disease and Related Dementias$\unicode{x2013}$ADRD), we extracted frame-aligned embeddings from different internal layers of wav2vec 2.0 or Whisper combined with DistilBERT or RoBERTa. Unimodal, EF and late fusion (LF) models were trained with a transformer classifier, optimized, and then evaluated across 10 seeds. Performance consistently peaked in mid encoder layers ($\sim$8$\unicode{x2013}$10), with the single best F1 at Whisper + RoBERTa layer 9 and the best log loss at Whisper + DistilBERT layer 10. Acoustic-only models consistently outperformed text-only variants. EF boosts discrimination for genuinely acoustic embeddings, whereas LF improves probability calibration. Layer choice critically shapes clinical multimodal synergy.

Figures (3)

• Figure 1: Block diagram of a pipeline for EF and training of classification models. $N$ -- number of tokens, $T$ -- number of frames, $D_\text{text}$ -- size of text vector representation, $D_\text{audio}$ -- size of audio vector representation.
• Figure 2: Log loss results of selected strategies depending on the acoustic model layer for Whisper + RoBERTa scenario.
• Figure 3: F1 score results of selected strategies depending on the acoustic model layer for Whisper + RoBERTa scenario.