Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environments

Sagarika Alavilli, Annesya Banerjee, Gasser Elbanna, Annika Magaro

TL;DR

The paper addresses the vulnerability of state-of-the-art ASR to noise and augmentations by proposing a dual-task transformer that jointly performs speech recognition and speaker identification using fused Whisper and ECAPA-TDNN embeddings. The approach matches Whisper on clean data and significantly outperforms it in noisy and heavily degraded conditions, including multi-talker babble and various augmentation schemes. By retaining speaker-related information rather than discarding it, the model gains robustness under adversarial environments, suggesting a practical path to more resilient speech recognition systems. Overall, incorporating talker identity cues enables better speaker tracking and recognition resilience in challenging acoustic contexts.

Abstract

Current state-of-the-art speech recognition models are trained to map acoustic signals into sub-lexical units. While these models demonstrate superior performance, they remain vulnerable to out-of-distribution conditions such as background noise and speech augmentations. In this work, we hypothesize that incorporating speaker representations during speech recognition can enhance model robustness to noise. We developed a transformer-based model that jointly performs speech recognition and speaker identification. Our model utilizes speech embeddings from Whisper and speaker embeddings from ECAPA-TDNN, which are processed jointly to perform both tasks. We show that the joint model performs comparably to Whisper under clean conditions. Notably, the joint model outperforms Whisper in high-noise environments, such as with 8-speaker babble background noise. Furthermore, our joint model excels in handling highly augmented speech, including sine-wave and noise-vocoded speech. Overall, these results suggest that integrating voice representations with speech recognition can lead to more robust models under adversarial conditions.

Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environments

TL;DR

The paper addresses the vulnerability of state-of-the-art ASR to noise and augmentations by proposing a dual-task transformer that jointly performs speech recognition and speaker identification using fused Whisper and ECAPA-TDNN embeddings. The approach matches Whisper on clean data and significantly outperforms it in noisy and heavily degraded conditions, including multi-talker babble and various augmentation schemes. By retaining speaker-related information rather than discarding it, the model gains robustness under adversarial environments, suggesting a practical path to more resilient speech recognition systems. Overall, incorporating talker identity cues enables better speaker tracking and recognition resilience in challenging acoustic contexts.

Abstract

Current state-of-the-art speech recognition models are trained to map acoustic signals into sub-lexical units. While these models demonstrate superior performance, they remain vulnerable to out-of-distribution conditions such as background noise and speech augmentations. In this work, we hypothesize that incorporating speaker representations during speech recognition can enhance model robustness to noise. We developed a transformer-based model that jointly performs speech recognition and speaker identification. Our model utilizes speech embeddings from Whisper and speaker embeddings from ECAPA-TDNN, which are processed jointly to perform both tasks. We show that the joint model performs comparably to Whisper under clean conditions. Notably, the joint model outperforms Whisper in high-noise environments, such as with 8-speaker babble background noise. Furthermore, our joint model excels in handling highly augmented speech, including sine-wave and noise-vocoded speech. Overall, these results suggest that integrating voice representations with speech recognition can lead to more robust models under adversarial conditions.
Paper Structure (11 sections, 1 equation, 4 figures, 1 table)

This paper contains 11 sections, 1 equation, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Candidate Models
  4. Experiments
  5. Datasets
  6. Results
  7. Speech and Speaker Recognition Performance
  8. 8 Speaker Babble Experiment
  9. Augmented Speech Experiment
  10. Discussion and Conclusion
  11. Acknowledgments

Figures (4)

  • Figure 1: Architecture of dual-task Joint model. The linear layer, ReLU activation, and mapping layer (to corresponding task logits), as described in the text, are represented together as the Linear Projection + ReLU stage. B, N, and H denote the batch size, the number of time frames, and the number of feed-forward nodes in the transformer layer, respectively.
  • Figure 2: Example spectrograms of an original speech excerpt, sine wave speech, and noise-vocoded speech. (Left) Original speech excerpt. (Middle) Sine wave speech, generated with four bands from the original speech excerpt. (Right) noise-vocoded speech, generated with four channels from the original speech excerpt.
  • Figure 3: Comparison of model speech recognition performance speech in 8-speaker babble. Character error rate is plotted from SNRs ranging from -15 dB to 20 dB in increments of 5, with the +inf label representing speech without noise.
  • Figure 4: Comparison of model speech recognition performance on two forms of augmented speech. (Left) Performance on noise-vocoded speech. (Right) Performance on sine wave speech.