Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Self-supervised Pre-training using Accent-Specific Codebooks

Darshan Prabhu, Abhishek Gupta, Omkar Nitsure, Preethi Jyothi, Sriram Ganapathy

TL;DR

This work tackles robust speech recognition across diverse accents by integrating accent-specific codebooks into a HuBERT-based self-supervised framework via cross-attention. The two-stage approach first pretrains an encoder with accent-aware codebooks and then fine-tunes with a joint CTC-attention objective, yielding significant WER reductions on both seen and unseen accents on the Mozilla Common Voice corpus and demonstrating zero-shot generalization to the L2-Arctic dataset. Key contributions include the design of deterministic accent-specific codebooks, their integration across encoder layers, and comprehensive ablations showing optimal layer placement and codebook size. The findings suggest that accent-informed SSL pretraining can substantially improve robustness to domain shifts in speech, with potential for further gains through semi-supervised self-training on unlabeled data.

Abstract

Speech accents present a serious challenge to the performance of state-of-the-art end-to-end Automatic Speech Recognition (ASR) systems. Even with self-supervised learning and pre-training of ASR models, accent invariance is seldom achieved. In this work, we propose an accent-aware adaptation technique for self-supervised learning that introduces a trainable set of accent-specific codebooks to the self-supervised architecture. These learnable codebooks enable the model to capture accent specific information during pre-training, that is further refined during ASR finetuning. On the Mozilla Common Voice dataset, our proposed approach outperforms all other accent-adaptation approaches on both seen and unseen English accents, with up to 9% relative reduction in word error rate (WER).

Improving Self-supervised Pre-training using Accent-Specific Codebooks

TL;DR

This work tackles robust speech recognition across diverse accents by integrating accent-specific codebooks into a HuBERT-based self-supervised framework via cross-attention. The two-stage approach first pretrains an encoder with accent-aware codebooks and then fine-tunes with a joint CTC-attention objective, yielding significant WER reductions on both seen and unseen accents on the Mozilla Common Voice corpus and demonstrating zero-shot generalization to the L2-Arctic dataset. Key contributions include the design of deterministic accent-specific codebooks, their integration across encoder layers, and comprehensive ablations showing optimal layer placement and codebook size. The findings suggest that accent-informed SSL pretraining can substantially improve robustness to domain shifts in speech, with potential for further gains through semi-supervised self-training on unlabeled data.

Abstract

Speech accents present a serious challenge to the performance of state-of-the-art end-to-end Automatic Speech Recognition (ASR) systems. Even with self-supervised learning and pre-training of ASR models, accent invariance is seldom achieved. In this work, we propose an accent-aware adaptation technique for self-supervised learning that introduces a trainable set of accent-specific codebooks to the self-supervised architecture. These learnable codebooks enable the model to capture accent specific information during pre-training, that is further refined during ASR finetuning. On the Mozilla Common Voice dataset, our proposed approach outperforms all other accent-adaptation approaches on both seen and unseen English accents, with up to 9% relative reduction in word error rate (WER).
Paper Structure (10 sections, 4 equations, 1 figure, 5 tables)

This paper contains 10 sections, 4 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Self-supervised Pre-training with Codebooks
  4. Supervised ASR Fine-tuning
  5. Inference using Codebooks
  6. Experimental Setup
  7. Experimental Results and Analysis
  8. Conclusion
  9. Acknowledgements
  10. References

Figures (1)

  • Figure 1: Overview of the two-stage training pipeline used in our proposed approach. In stage 1, we train an Encoder only model with the HuBERT style pre-training objective. Our architecture incorporates Accent-specific codebooks and additional Cross-Attention layers. The model trained in this stage is used as Encoder in stage 2, where we train an Encoder-Decoder model with Joint CTC-Attention objective for ASR fine-tuning.