Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient infusion of self-supervised representations in Automatic Speech Recognition

Darshan Prabhu, Sai Ganesh Mirishkar, Pankaj Wasnik

TL;DR

The paper tackles leveraging self-supervised speech representations within ASR without retraining large SSL encoders. It introduces two fusion mechanisms, Subsampled Framewise Addition (SFA) and Cross Attention (CA), to inject SSL features into a Conformer-based encoder while keeping the model compact. Empirical results on Librispeech-100 and Tedlium2 show that CA, particularly with HuBERT embeddings, yields significant WER improvements and faster convergence, with only modest preprocessing overhead from pre-extracted SSL features. Ablations and attention visualizations reveal learnable frame alignments and a favorable trade-off between accuracy, training speed, and parameter efficiency.

Abstract

Self-supervised learned (SSL) models such as Wav2vec and HuBERT yield state-of-the-art results on speech-related tasks. Given the effectiveness of such models, it is advantageous to use them in conventional ASR systems. While some approaches suggest incorporating these models as a trainable encoder or a learnable frontend, training such systems is extremely slow and requires a lot of computation cycles. In this work, we propose two simple approaches that use (1) framewise addition and (2) cross-attention mechanisms to efficiently incorporate the representations from the SSL model(s) into the ASR architecture, resulting in models that are comparable in size with standard encoder-decoder conformer systems while also avoiding the usage of SSL models during training. Our approach results in faster training and yields significant performance gains on the Librispeech and Tedlium datasets compared to baselines. We further provide detailed analysis and ablation studies that demonstrate the effectiveness of our approach.

Efficient infusion of self-supervised representations in Automatic Speech Recognition

TL;DR

The paper tackles leveraging self-supervised speech representations within ASR without retraining large SSL encoders. It introduces two fusion mechanisms, Subsampled Framewise Addition (SFA) and Cross Attention (CA), to inject SSL features into a Conformer-based encoder while keeping the model compact. Empirical results on Librispeech-100 and Tedlium2 show that CA, particularly with HuBERT embeddings, yields significant WER improvements and faster convergence, with only modest preprocessing overhead from pre-extracted SSL features. Ablations and attention visualizations reveal learnable frame alignments and a favorable trade-off between accuracy, training speed, and parameter efficiency.

Abstract

Self-supervised learned (SSL) models such as Wav2vec and HuBERT yield state-of-the-art results on speech-related tasks. Given the effectiveness of such models, it is advantageous to use them in conventional ASR systems. While some approaches suggest incorporating these models as a trainable encoder or a learnable frontend, training such systems is extremely slow and requires a lot of computation cycles. In this work, we propose two simple approaches that use (1) framewise addition and (2) cross-attention mechanisms to efficiently incorporate the representations from the SSL model(s) into the ASR architecture, resulting in models that are comparable in size with standard encoder-decoder conformer systems while also avoiding the usage of SSL models during training. Our approach results in faster training and yields significant performance gains on the Librispeech and Tedlium datasets compared to baselines. We further provide detailed analysis and ablation studies that demonstrate the effectiveness of our approach.
Paper Structure (14 sections, 3 figures, 3 tables)

This paper contains 14 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Methodology
  3. Fusion Layer
  4. Subsampled Framewise Addition (SFA):
  5. Cross Attention (CA):
  6. Experimental Setup
  7. Results
  8. Analysis
  9. Faster Model Convergence:
  10. Efficacy of our approach:
  11. Number of encoder layers:
  12. Visualization of Attention:
  13. Performance on unseen dataset:
  14. Conclusion

Figures (3)

  • Figure 1: Overview of our proposed Architecture that integrates the representations from the self-supervised model into the ASR encoder using two approaches: Subsampled Framewise Addition(SFA) and Cross Attention(CA). $\hat{v}$ and $\hat{u}$ are the self-supervised and fbank representations respectively.
  • Figure 2: Comparison of epochwise model performance(WER) between baseline and our best setting( Conf.+HuBERT-Base+CA) on the validation split of Librispeech-100 dataset.
  • Figure 3: Visualization of attention scores for two samples from the Librispeech-100 dataset, one from Dev-Other split and Test-Other split respectively. We use the model from our best setting to obtain these scores.