Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Probing self-attention in self-supervised speech models for cross-linguistic differences

Sai Gopinath, Joselyn Rodriguez

TL;DR

This paper finds that even with a small model, the attention heads learned are diverse ranging from almost entirely diagonal to almost entirely global regardless of the training language, and presents a head ablation study which shows that models across languages primarily rely on diagonal heads to classify phonemes.

Abstract

Speech models have gained traction thanks to increase in accuracy from novel transformer architectures. While this impressive increase in performance across automatic speech recognition (ASR) benchmarks is noteworthy, there is still much that is unknown about the use of attention mechanisms for speech-related tasks. For example, while it is assumed that these models are learning language-independent (i.e., universal) speech representations, there has not yet been an in-depth exploration of what it would mean for the models to be language-independent. In the current paper, we explore this question within the realm of self-attention mechanisms of one small self-supervised speech transformer model (TERA). We find that even with a small model, the attention heads learned are diverse ranging from almost entirely diagonal to almost entirely global regardless of the training language. We highlight some notable differences in attention patterns between Turkish and English and demonstrate that the models do learn important phonological information during pretraining. We also present a head ablation study which shows that models across languages primarily rely on diagonal heads to classify phonemes.

Probing self-attention in self-supervised speech models for cross-linguistic differences

TL;DR

This paper finds that even with a small model, the attention heads learned are diverse ranging from almost entirely diagonal to almost entirely global regardless of the training language, and presents a head ablation study which shows that models across languages primarily rely on diagonal heads to classify phonemes.

Abstract

Speech models have gained traction thanks to increase in accuracy from novel transformer architectures. While this impressive increase in performance across automatic speech recognition (ASR) benchmarks is noteworthy, there is still much that is unknown about the use of attention mechanisms for speech-related tasks. For example, while it is assumed that these models are learning language-independent (i.e., universal) speech representations, there has not yet been an in-depth exploration of what it would mean for the models to be language-independent. In the current paper, we explore this question within the realm of self-attention mechanisms of one small self-supervised speech transformer model (TERA). We find that even with a small model, the attention heads learned are diverse ranging from almost entirely diagonal to almost entirely global regardless of the training language. We highlight some notable differences in attention patterns between Turkish and English and demonstrate that the models do learn important phonological information during pretraining. We also present a head ablation study which shows that models across languages primarily rely on diagonal heads to classify phonemes.
Paper Structure (3 sections, 3 equations, 7 figures, 3 tables)

This paper contains 3 sections, 3 equations, 7 figures, 3 tables.

Table of Contents

  1. Appendix
  2. Supplemental material
  3. License Declarations

Figures (7)

  • Figure 1: Examples of single Turkish utterance heads from each category for Turkish pre-trained TERA. Both axis correspond to frames.
  • Figure 2: Examples of single English utterance heads from each category for English pre-trained TERA. Both axis correspond to frames.
  • Figure 3: Number of Heads in Each Category for Each Language
  • Figure 4: English vs Turkish Heads and score values in each category. The y axis corresponds to the head name (layer number, head number), the columns are the languages, and the values are the scores for each head according to the category formulas. A blank space means that that head fell into a given category (diagonal, vertical, global) for one language but not the other
  • Figure 5: Phoneme relation maps for English (left) and Turkish (right) with vowels identified in blue box
  • ...and 2 more figures