Do Models Hear Like Us? Probing the Representational Alignment of Audio LLMs and Naturalistic EEG

TL;DR

The paper tackles whether Audio LLM representations align with human neural dynamics during naturalistic listening. It introduces a multi-metric, layer-wise EEG–Audio LLM alignment framework, combining RSA-based geometry with nonlinear dependence metrics and a novel Tri-modal Neighborhood Consistency criterion to incorporate acoustic, neural, and model representations, including prosodic factors. The results reveal depth- and metric-dependent alignment patterns, with strongest RSA in the 250–500 ms window (N400 range) and a robust rank–dependence split across models, highlighting how affective prosody can differentially modulate geometry versus covariance-based coupling. Together, these findings advance neurobiological insights into Audio LLM representations and provide a comprehensive, open framework for brain–model alignment applicable to naturalistic speech research and brain-aware language technologies.

Abstract

Audio Large Language Models (Audio LLMs) have demonstrated strong capabilities in integrating speech perception with language understanding. However, whether their internal representations align with human neural dynamics during naturalistic listening remains largely unexplored. In this work, we systematically examine layer-wise representational alignment between 12 open-source Audio LLMs and Electroencephalogram (EEG) signals across 2 datasets. Specifically, we employ 8 similarity metrics, such as Spearman-based Representational Similarity Analysis (RSA), to characterize within-sentence representational geometry. Our analysis reveals 3 key findings: (1) we observe a rank-dependence split, in which model rankings vary substantially across different similarity metrics; (2) we identify spatio-temporal alignment patterns characterized by depth-dependent alignment peaks and a pronounced increase in RSA within the 250-500 ms time window, consistent with N400-related neural dynamics; (3) we find an affective dissociation whereby negative prosody, identified using a proposed Tri-modal Neighborhood Consistency (TNC) criterion, reduces geometric similarity while enhancing covariance-based dependence. These findings provide new neurobiological insights into the representational mechanisms of Audio LLMs.

Figures (12)

• Figure 1: The motivation of this paper: exploring the similarity of human brain EEG signals and Audio LLMs representations through various similarity metrics.
• Figure 2: The workflow for EEG–Audio LLM similarity analysis. EEG and Audio LLM representations are preprocessed and temporally aligned, compared using 8 complementary similarity measures, and further analyzed with a tri-modal neighborhood consistency (TNC) metric for affective and prosodic alignment.
• Figure 3: Spatiotemporal dynamics of topographic similarity between Audio LLM and EEG signals.
• Figure 4: The heatmap results between electrodes and audio LLM layers.
• Figure 5: Paired representational dissimilarity matrices for audio model and EEG.

Theorems & Definitions (3)

• Proposition 1: Range
• Proposition 2: Near-1 TNC forces all three pairwise RSA scores to be near-1
• Proposition 3: Pair-specific confounds inflate at most one term of TNC