Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disentanglement of Sources in a Multi-Stream Variational Autoencoder

Veranika Boukun, Jörg Lücke

TL;DR

MS-VAE addresses disentangled source representation by introducing discrete latent indicators that select among multiple per-source VAEs connected through an explicit linear mixing model. The framework integrates discrete and continuous latents within a variational objective, deriving an ELBO with an energy-based encoder for the discrete factors and standard reparameterization for the continuous ones. Empirically, it demonstrates clear source separation on MNIST mixtures and competitive speaker diarization, achieving fewer missed speech frames with modest supervision and data requirements. Overall, the domain-agnostic MS-VAE highlights the value of explicit source-mixing constraints for interpretable and controllable disentangled representations in multi-source data.

Abstract

Variational autoencoders (VAEs) are a leading approach to address the problem of learning disentangled representations. Typically a single VAE is used and disentangled representations are sought in its continuous latent space. Here we explore a different approach by using discrete latents to combine VAE-representations of individual sources. The combination is done based on an explicit model for source combination, and we here use a linear combination model which is well suited, e.g., for acoustic data. We formally define such a multi-stream VAE (MS-VAE) approach, derive its inference and learning equations, and we numerically investigate its principled functionality. The MS-VAE is domain-agnostic, and we here explore its ability to separate sources into different streams using superimposed hand-written digits, and mixed acoustic sources in a speaker diarization task. We observe a clear separation of digits, and on speaker diarization we observe an especially low rate of missed speakers. Numerical experiments further highlight the flexibility of the approach across varying amounts of supervision and training data.

Disentanglement of Sources in a Multi-Stream Variational Autoencoder

TL;DR

MS-VAE addresses disentangled source representation by introducing discrete latent indicators that select among multiple per-source VAEs connected through an explicit linear mixing model. The framework integrates discrete and continuous latents within a variational objective, deriving an ELBO with an energy-based encoder for the discrete factors and standard reparameterization for the continuous ones. Empirically, it demonstrates clear source separation on MNIST mixtures and competitive speaker diarization, achieving fewer missed speech frames with modest supervision and data requirements. Overall, the domain-agnostic MS-VAE highlights the value of explicit source-mixing constraints for interpretable and controllable disentangled representations in multi-source data.

Abstract

Variational autoencoders (VAEs) are a leading approach to address the problem of learning disentangled representations. Typically a single VAE is used and disentangled representations are sought in its continuous latent space. Here we explore a different approach by using discrete latents to combine VAE-representations of individual sources. The combination is done based on an explicit model for source combination, and we here use a linear combination model which is well suited, e.g., for acoustic data. We formally define such a multi-stream VAE (MS-VAE) approach, derive its inference and learning equations, and we numerically investigate its principled functionality. The MS-VAE is domain-agnostic, and we here explore its ability to separate sources into different streams using superimposed hand-written digits, and mixed acoustic sources in a speaker diarization task. We observe a clear separation of digits, and on speaker diarization we observe an especially low rate of missed speakers. Numerical experiments further highlight the flexibility of the approach across varying amounts of supervision and training data.
Paper Structure (13 sections, 17 equations, 4 figures, 2 tables)

This paper contains 13 sections, 17 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Computation of the Encoder Expectations
  4. Optimization of the Encoding and Decoding Models
  5. Discrete Case Prediction
  6. Experiments
  7. Neural networks and hyperparameter settings
  8. Proof of Concept
  9. Extension to multiple sources
  10. Application to Acoustic Data
  11. Conclusion
  12. Compliance with Ethical Standards
  13. Acknowledgments

Figures (4)

  • Figure 1: Schematic representation of MS-VAE for $K=2$.
  • Figure 2: (a) Examples of generated artificial data points for subset of digits '0' and '1'. (b) Schematic of source separation results.
  • Figure 3: Source disentanglement using the full MNIST dataset with 10% labels of available labels. Images 1 shows the input and image 2 shows the MS-VAE reconstruction; the remaining ten images correspond to the reconstructions that the MS-VAE internally uses for input reconstruction. Finetuning the model lead to slightly better reconstruction, see Tab. \ref{['tab:mnist_results']}.
  • Figure 4: Spectrograms of original/estimated mixtures with decision boundaries indicating case changes, e.g., $0 \rightarrow 2$: silence to speaker 1, etc., similarly to Fig. \ref{['fig:schematic-results']}(a). Red: ground truth; green: predicted (Eq. \ref{['eq:q_s_x']}) averaged over $1s$ intervals.