Focused Discriminative Training For Streaming CTC-Trained Automatic Speech Recognition Models
Adnan Haider, Xingyu Na, Erik McDermott, Tim Ng, Zhen Huang, Xiaodan Zhuang
TL;DR
This work introduces Focused Discriminative Training (FDT), a lattice-free, HMM-independent framework to further improve streaming word-piece ASR models trained with CTC or CTC+AED by concentrating discriminative effort on challenging audio segments. FDT combines implicit data selection, error-region detection, and a constrained CTC-based segment-level contrastive loss to selectively fine-tune the model on difficult portions of the input, guided by an EM-like optimization. Empirically, FDT provides larger WER reductions than traditional discriminative training methods (MMI, MWER) on LibriSpeech and also yields gains when applied to a converged, large-scale (600k hours) domain-adapted model, while incurring lower training overhead due to its lattice-free design. The results also illuminate how AED components influence model entropy and how segmentation constraints affect gains, outlining clear directions for extending FDT to broader streaming and non-streaming settings and data regimes.
Abstract
This paper introduces a novel training framework called Focused Discriminative Training (FDT) to further improve streaming word-piece end-to-end (E2E) automatic speech recognition (ASR) models trained using either CTC or an interpolation of CTC and attention-based encoder-decoder (AED) loss. The proposed approach presents a novel framework to identify and improve a model's recognition on challenging segments of an audio. Notably, this training framework is independent of hidden Markov models (HMMs) and lattices, eliminating the need for substantial decision-making regarding HMM topology, lexicon, and graph generation, as typically required in standard discriminative training approaches. Compared to additional fine-tuning with MMI or MWER loss on the encoder, FDT is shown to be more effective in achieving greater reductions in Word Error Rate (WER) on streaming models trained on LibriSpeech. Additionally, this method is shown to be effective in further improving a converged word-piece streaming E2E model trained on 600k hours of assistant and dictation dataset.