Bayesian Learning for Deep Neural Network Adaptation

TL;DR

This work tackles speaker mismatch in ASR by introducing a full Bayesian framework for deep neural network speaker adaptation to model SD parameter uncertainty when adaptation data are scarce. It applies the Bayesian treatment to three structured transforms—LHUC, HUB, and PAct—resulting in BLHUC, BHUB, and BPAct, with efficient variational inference and test-time sampling techniques. Across 300-hour Switchboard experiments and augmented 900-hour data, Bayesian adaptations consistently outperform their deterministic counterparts and, in several settings, approach or surpass state-of-the-art results, including strong performance after limited adaptation data. The approach offers rapid, robust adaptation suitable for real-world applications, with code made available for replication and further exploration.

Abstract

A key task for speech recognition systems is to reduce the mismatch between training and evaluation data that is often attributable to speaker differences. Speaker adaptation techniques play a vital role to reduce the mismatch. Model-based speaker adaptation approaches often require sufficient amounts of target speaker data to ensure robustness. When the amount of speaker level data is limited, speaker adaptation is prone to overfitting and poor generalization. To address the issue, this paper proposes a full Bayesian learning based DNN speaker adaptation framework to model speaker-dependent (SD) parameter uncertainty given limited speaker specific adaptation data. This framework is investigated in three forms of model based DNN adaptation techniques: Bayesian learning of hidden unit contributions (BLHUC), Bayesian parameterized activation functions (BPAct), and Bayesian hidden unit bias vectors (BHUB). In the three methods, deterministic SD parameters are replaced by latent variable posterior distributions for each speaker, whose parameters are efficiently estimated using a variational inference based approach. Experiments conducted on 300-hour speed perturbed Switchboard corpus trained LF-MMI TDNN/CNN-TDNN systems suggest the proposed Bayesian adaptation approaches consistently outperform the deterministic adaptation on the NIST Hub5'00 and RT03 evaluation sets. When using only the first five utterances from each speaker as adaptation data, significant word error rate reductions up to 1.4% absolute (7.2% relative) were obtained on the CallHome subset. The efficacy of the proposed Bayesian adaptation techniques is further demonstrated in a comparison against the state-of-the-art performance obtained on the same task using the most recent systems reported in the literature.

Figures (4)

• Figure 1: Examples of applying learning hidden unit contributions (LHUC) adaptation (left), hidden unit bias vectors (HUB) adaptation (middle), and parameterized activation functions (PAct) adaptation (right) to a hidden layer of DNN acoustic model.
• Figure 2: Examples of applying Bayesian LHUC (BLHUC) adaptation (left), Bayesian hidden unit bias vectors (BHUB) adaptation (middle), and Bayesian parameterized activation functions (BPAct) adaptation (right) to a hidden layer of DNN acoustic model.
• Figure 3: Performance contrast of applying LHUC and BLHUC adaptation to varying number of hidden layers and using different activation functions for LHUC scaling vectors, including 2Sigmoid, identity, and exponential. The evaluation was conducted on the CHE subset of Hub5'00. The BLHUC adaptation was repeated 10 times with different random seeds (for sampling in equation (\ref{['eq:integal']})) to show the average performance in word error rate (WER) and standard deviation.
• Figure 4: Performance in WER obtained by different adaptation techniques using different learning rates and various amounts of adaptation data, based on the systems using i-vector adaptation in Table \ref{['tab:amount']} (Systems (9) to (20)), including LHUC/BLHUC, HUB/BHUB, PAct/BPAct, and their SAT versions. The evaluation was conducted on the CHE subset of Hub5'00. The name of a curve, for example, "LHUC with 5 utts", means the results obtained by LHUC adaptation using 5 utterances as adaptation data. Among these curves, the "$\times$" denotes the results of deterministic adaptation techniques, the "$\circ$" denotes those of Bayesian adaptation techniques, and curves in different colour denote using different amounts of adaptation data. The empirically adjusted and fixed learning rates used for different adaptation methods (0.01 for LHUC/BLHUC and PAct/BPAct, and 0.000001 for HUB/BHUB) are denoted by the numbers in red colour on the "Learning Rate" axis.