Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do self-supervised speech and language models extract similar representations as human brain?

Peili Chen, Linyang He, Li Fu, Lu Fan, Edward F. Chang, Yuanning Li

TL;DR

The paper investigates whether self-supervised speech and language models extract representations that align with the human brain during speech perception. It directly compares Wav2Vec2.0 and GPT-2 by predicting electrocorticography responses using a temporal receptive field encoder and probing shared information with Canonical Correlation Analysis. The findings show both models predict auditory cortex activity and share substantial variance, dominated by contextual information, with Wav2Vec2.0 contributing additional unique content. This suggests convergence of contextual representations in SSL models with the brain's speech perception network and provides insights for both SSL models and neuroscience of speech and language processing.

Abstract

Speech and language models trained through self-supervised learning (SSL) demonstrate strong alignment with brain activity during speech and language perception. However, given their distinct training modalities, it remains unclear whether they correlate with the same neural aspects. We directly address this question by evaluating the brain prediction performance of two representative SSL models, Wav2Vec2.0 and GPT-2, designed for speech and language tasks. Our findings reveal that both models accurately predict speech responses in the auditory cortex, with a significant correlation between their brain predictions. Notably, shared speech contextual information between Wav2Vec2.0 and GPT-2 accounts for the majority of explained variance in brain activity, surpassing static semantic and lower-level acoustic-phonetic information. These results underscore the convergence of speech contextual representations in SSL models and their alignment with the neural network underlying speech perception, offering valuable insights into both SSL models and the neural basis of speech and language processing.

Do self-supervised speech and language models extract similar representations as human brain?

TL;DR

The paper investigates whether self-supervised speech and language models extract representations that align with the human brain during speech perception. It directly compares Wav2Vec2.0 and GPT-2 by predicting electrocorticography responses using a temporal receptive field encoder and probing shared information with Canonical Correlation Analysis. The findings show both models predict auditory cortex activity and share substantial variance, dominated by contextual information, with Wav2Vec2.0 contributing additional unique content. This suggests convergence of contextual representations in SSL models with the brain's speech perception network and provides insights for both SSL models and neuroscience of speech and language processing.

Abstract

Speech and language models trained through self-supervised learning (SSL) demonstrate strong alignment with brain activity during speech and language perception. However, given their distinct training modalities, it remains unclear whether they correlate with the same neural aspects. We directly address this question by evaluating the brain prediction performance of two representative SSL models, Wav2Vec2.0 and GPT-2, designed for speech and language tasks. Our findings reveal that both models accurately predict speech responses in the auditory cortex, with a significant correlation between their brain predictions. Notably, shared speech contextual information between Wav2Vec2.0 and GPT-2 accounts for the majority of explained variance in brain activity, surpassing static semantic and lower-level acoustic-phonetic information. These results underscore the convergence of speech contextual representations in SSL models and their alignment with the neural network underlying speech perception, offering valuable insights into both SSL models and the neural basis of speech and language processing.
Paper Structure (10 sections, 2 equations, 4 figures)

This paper contains 10 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. ANALYSIS METHODS
  3. Overview of the analysis
  4. Deep neural encoding model using SSL speech and language models
  5. Brain-prediction performance metric
  6. Analyze shared feature representations using Canonical Correlation Analysis
  7. Experiment results
  8. Experiment setup
  9. Analysis results
  10. Conclusion

Figures (4)

  • Figure 1: Predicting brain activity using SSL models.
  • Figure 2: a. Brain prediction scores of Wav2Vec2.0 and GPT-2 on the left and right brain hemispheres, each dot is a single ECoG electrode. b. Average normalized prediction score over the electrodes in brain areas related to speech perception, compared to a baseline prediction model using mel spectrum features. (P values computed from paired t-test, Bonferroni correction)
  • Figure 3: a. An example of the actual and SSL-predicted neural activity corresponding to a sentence from one ECoG electrode in STG. b. The similarity between the brain activity predictions of Wav2Vec2.0 (w2v2), GPT-2 (gpt2) and mel spectrum (mel), quantified by shared variance ($R^2$). (P values computed from paired t-test, Bonferroni correction)
  • Figure 4: a. Normalized prediction score of the top 50 canonical variables (*CCs) and the remaining canonical variables (rest *CCs) between Wav2Vec2.0 (*=w2v2) and GPT-2 (*=gpt2), compared to their original prediction scores (Wav2Vec2.0 and GPT-2), and mel-spectrum baseline (dashed line). b. The brain prediction performance computed from the decomposition of the shared CCs (gpt2CCs). The decomposition was computed from CCA between gpt2CCs and three models: GloVe (static semantic CCs), mel spectrum (Mel Spec CCs) and residual context (Contextual CCs). (P values computed from paired t-test, Bonferroni correction)