Self-consistent context aware conformer transducer for speech recognition
Konstantin Kolokolov, Pavel Pekichev, Karthik Raghunathan
TL;DR
The paper tackles rare-word recognition in end-to-end ASR by introducing a self-consistent recursive module that injects external contextual cues into a conformer transducer. The module iteratively solves a coupled set of equations binding the audio-path and context-path, typically converging within $3$ iterations. Embedded in a context-aware transducer and combined with shallow fusion against a context language model, it improves OOV accuracy substantially while keeping WER changes small. The approach is compatible with multiple encoders (e.g., SqueezeFormer, Fast Conformer, Zipformer) and can generalize to other recursive data-flow tasks, suggesting broad applicability beyond ASR.
Abstract
We introduce a novel neural network module that adeptly handles recursive data flow in neural network architectures. At its core, this module employs a self-consistent approach where a set of recursive equations is solved iteratively, halting when the difference between two consecutive iterations falls below a defined threshold. Leveraging this mechanism, we construct a new neural network architecture, an extension of the conformer transducer, which enriches automatic speech recognition systems with a stream of contextual information. Our method notably improves the accuracy of recognizing rare words without adversely affecting the word error rate for common vocabulary. We investigate the improvement in accuracy for these uncommon words using our novel model, both independently and in conjunction with shallow fusion with a context language model. Our findings reveal that the combination of both approaches can improve the accuracy of detecting rare words by as much as 4.5 times. Our proposed self-consistent recursive methodology is versatile and adaptable, compatible with many recently developed encoders, and has the potential to drive model improvements in speech recognition and beyond.