Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory tasks

TL;DR

The study shows that brain-like audio representations emerge as self-supervised models improve on diverse downstream tasks, with stronger brain alignment corresponding to better performance on HEAREval benchmarks. By analyzing 36 audio models against two fMRI datasets using voxel-wise regression, RSA, and component analyses, the authors demonstrate that data diversity and self-supervised pretraining drive convergence toward auditory cortex representations. Importantly, brain alignment appears early during pretraining, suggesting alignment is an emergent byproduct of learning to reconstruct missing audio information. These findings support the Platonic Representation Hypothesis and have practical implications for using brain-alignment as a proxy to evaluate and guide audio representation learning.

Abstract

Artificial neural networks (ANNs) are increasingly powerful models of brain computation, yet it remains unclear whether improving their task performance also makes their internal representations more similar to brain signals. To address this question in the auditory domain, we quantified the alignment between the internal representations of 36 different audio models and brain activity from two independent fMRI datasets. Using voxel-wise and component-wise regression, and representation similarity analysis (RSA), we found that recent self-supervised audio models with strong performance in diverse downstream tasks are better predictors of auditory cortex activity than older and more specialized models. To assess the quality of the audio representations, we evaluated these models in 6 auditory tasks from the HEAREval benchmark, spanning music, speech, and environmental sounds. This revealed strong positive Pearson correlations ($r>0.7$) between a model's overall task performance and its alignment with brain representations. Finally, we analyzed the evolution of the similarity between audio and brain representations during the pretraining of EnCodecMAE. We discovered that brain similarity increases progressively and emerges early during pretraining, despite the model not being explicitly optimized for this objective. This suggests that brain-like representations can be an emergent byproduct of learning to reconstruct missing information from naturalistic audio data.

Figures (12)

• Figure 1: Schematic depicting the two main analysis we performed to compare audio and brain representations: regression analysis and representation similarity analysis (RSA). For regression, the target variable is the fMRI activity $Y_{s, v}$ of a voxel $v$ and subject $s$ and the predictor variable is the activation map $X_{m,l}$ from layer $l$ of an audio model $m$. For RSA, RDM matrices are calculated from $X_{m,l}$ and all the voxels from a subject $Y_{s}$ and compared using Spearman Correlation.
• Figure 2: Left: $R^2$ obtained for the analyzed audio models. Right: same results as in left, selecting groups of systems to highlight various interesting conclusions discussed in the text. The gray line corresponds to a spectro-temporal baseline system, and error bars reflect the standard error measured across subjects.
• Figure 3: Example of an RDM computed from NH2015 fMRI responses (left) and from EnCodecMAE Large (right). Rows and columns correspond to the 165 auditory stimuli, grouped by stimulus type as indicated by colors.
• Figure 4: RSA coefficient, $\rho_m$, given by the Spearman correlation between brain and model RDMs. Gray lines correspond to the spectro-temporal baseline and the inter-subject RSA topline. Error bars reflect the standard error across subjects.
• Figure 5: $R^2$ values for each of the six components and the audio models evaluated. The gray lines correspond to the spectro-temporal baseline.