Variational Bayesian Adaptive Learning of Deep Latent Variables for Acoustic Knowledge Transfer

TL;DR

This work tackles acoustic mismatches across domains (e.g., devices and noise) by transferring knowledge through a compact set of deep latent variables $Z$ rather than adapting millions of parameters. It introduces a variational Bayesian adaptive learning (VBAL) framework with two posterior-estimation strategies: Gaussian mean-field variational inference (GMFVI) when parallel source-target data exist, and empirical Bayes (EB) when such data are unavailable, augmented by structural relationship modeling to better capture latent dependencies. The approach is validated on acoustic scene classification (ASC) for device adaptation and spoken command recognition (SCR) under noise, showing consistent improvements over state-of-the-art knowledge-transfer methods and several baselines. By encoding source-domain knowledge in latent-variable priors and combining it with limited target data via variational inference, the method achieves data-efficient cross-domain adaptation with strong empirical performance and interpretable latent representations. The framework lays groundwork for broader latent-variable-based cross-domain transfer in audio, suggesting avenues for exploring alternative Bayesian priors and inference schemes.

Abstract

In this work, we propose a novel variational Bayesian adaptive learning approach for cross-domain knowledge transfer to address acoustic mismatches between training and testing conditions, such as recording devices and environmental noise. Different from the traditional Bayesian approaches that impose uncertainties on model parameters risking the curse of dimensionality due to the huge number of parameters, we focus on estimating a manageable number of latent variables in deep neural models. Knowledge learned from a source domain is thus encoded in prior distributions of deep latent variables and optimally combined, in a Bayesian sense, with a small set of adaptation data from a target domain to approximate the corresponding posterior distributions. Two different strategies are proposed and investigated to estimate the posterior distributions: Gaussian mean-field variational inference, and empirical Bayes. These strategies address the presence or absence of parallel data in the source and target domains. Furthermore, structural relationship modeling is investigated to enhance the approximation. We evaluated our proposed approaches on two acoustic adaptation tasks: 1) device adaptation for acoustic scene classification, and 2) noise adaptation for spoken command recognition. Experimental results show that the proposed variational Bayesian adaptive learning approach can obtain good improvements on target domain data, and consistently outperforms state-of-the-art knowledge transfer methods.

Figures (5)

• Figure 1: Illustration of the deep latent variables based acoustic knowledge transfer framework.
• Figure 2: Histograms of sample hidden embedding elements from 5 classes. This visualization is performed using the CRNN-att model of the SCR task. Each subplot represents an element of the hidden embedding, as each one is accumulated by class index, where subplots in the same row correspond to the same class.
• Figure 3: Evaluation results (classification accuracy %) of using different hidden layers by FCNN model for ASC task. Two target devices are shown: (a) Device s3 and (b) Device s5. The basic TSL is combined with all methods.
• Figure 4: Visualized heatmaps of the intra-class discrepancy between target outputs. Each cell in the subnets illustrates the level of discrepancy between two outputs, where darker colors indicate greater intra-class discrepancies.
• Figure 5: Feature visualization via T-SNE results of four devices on ASC task. Each point in the scatter plots corresponds to a hidden feature generated from the audio sample.