Transducer Consistency Regularization for Speech to Text Applications

TL;DR

This study presents Transducer Consistency Regularization (TCR), a consistency regularization method for transducer models that could effectively reduce word error rate (WER) by 4.3% relatively comparing with a strong baseline on the Librispeech dataset.

Abstract

Consistency regularization is a commonly used practice to encourage the model to generate consistent representation from distorted input features and improve model generalization. It shows significant improvement on various speech applications that are optimized with cross entropy criterion. However, it is not straightforward to apply consistency regularization for the transducer-based approaches, which are widely adopted for speech applications due to the competitive performance and streaming characteristic. The main challenge is from the vast alignment space of the transducer optimization criterion and not all the alignments within the space contribute to the model optimization equally. In this study, we present Transducer Consistency Regularization (TCR), a consistency regularization method for transducer models. We apply distortions such as spec augmentation and dropout to create different data views and minimize the distribution difference. We utilize occupational probabilities to give different weights on transducer output distributions, thus only alignments close to oracle alignments would contribute to the model learning. Our experiments show the proposed method is superior to other consistency regularization implementations and could effectively reduce word error rate (WER) by 4.3\% relatively comparing with a strong baseline on the \textsc{Librispeech} dataset.

Figures (4)

• Figure 1: Transducer model. Composed of encoder, predictor, and joiner. The encoder transforms current audio data into hidden audio representation, predictor receives the previous token and predict the hidden text representation, and joiner takes the two hidden representations to predicts the current token.
• Figure 2: TCR flow chart. The data is first duplicated, then distorted with spec augmentation and dropout to create two different views of data representation. Finally KL divergence is used to minimize the representation difference.
• Figure 3: Transducer decoding lattice. Red horizontal arrow represents blank token emission, while green vertical arrow represents non blank token emission. One full path starts from $(t, u) = (1, 0)$ and ends at $(t, u) = (T, U)$.
• Figure 4: Occupational probability of the same audio with two different views. The occupational probability is calculated using the forward variable $\alpha$ and backward variable $\beta$. The more concentrated the blue color, the higher the occupational probability. Only a few paths have non-zero occupational probability. The two paths of different data views mostly agree except on the horizontal and vertical transitions.