End-to-End Target Speaker Speech Recognition Using Context-Aware Attention Mechanisms for Challenging Enrollment Scenario

TL;DR

The paper tackles robust end-to-end target-speaker ASR in streaming settings with noisy and overlapping enrollment and flexible enrollment phrases. It extends the TS-RNNT framework into RobustTS-RNNT by introducing dynamic contextual biasing and a text-guided attention mechanism, enabling frame-specific speaker representation and semi-text-dependent enrollment guided by wake words. On a synthesized Persian dataset combining DeepMine and WHAM! noise, RobustTS-RNNT achieves a WER of $16.44\%$ at $-5$ dB SIR with overlapping enrollment, a substantial improvement over baselines that exceed $75\%$ WER under the same conditions. This demonstrates strong robustness and practical potential for voice-controlled devices in realistic noisy environments.

Abstract

This paper presents a novel streaming end-to-end target-speaker speech recognition that addresses two critical limitations in systems: the handling of noisy enrollment utterances and specific enrollment phrase requirements. This paper proposes a robust Target-Speaker Recurrent Neural Network Transducer (TS-RNNT) with dual attention mechanisms for contextual biasing and overlapping enrollment processing. The model incorporates a text decoder and attention mechanism specifically designed to extract relevant speaker characteristics from noisy, overlapping enrollment audio. Experimental results on a synthesized dataset demonstrate the model's resilience, maintaining a Word Error Rate (WER) of 16.44% even with overlapping enrollment at 5dB Signal-to-Interference Ratio (SIR), compared to conventional approaches that degrade to WERs above 75% under similar conditions. This significant performance improvement, coupled with the model's semi-text-dependent enrollment capabilities, represents a substantial advancement toward more practical and versatile voice-controlled devices.

Figures (1)

• Figure 1: Architectures of the Target-Speaker RNN Transducer (TS-RNNT). In (a), the TS-RNNT architecture includes standard RNNT components (blue), modules that capture target speaker characteristics (yellow), and speaker-specific bias (white). In (b), the RobustTS-RNNT architecture extends TS-RNNT by incorporating Contextual Biasing and a Text Attention Mechanism, where additional yellow modules capture target speaker features with text information. Model weights in RobustTS-RNNT are initialized from the offline Zipformer (yellow) and streaming Zipformer (blue).