Variational decomposition autoencoding improves disentanglement of latent representations

TL;DR

Variational decomposition autoencoding (VDA), a framework that extends VAEs by incorporating a strong structural bias toward signal decomposition, is introduced, suggesting that decomposition-aware architectures can serve as robust tools for extracting structured representations from dynamic signals.

Abstract

Understanding the structure of complex, nonstationary, high-dimensional time-evolving signals is a central challenge in scientific data analysis. In many domains, such as speech and biomedical signal processing, the ability to learn disentangled and interpretable representations is critical for uncovering latent generative mechanisms. Traditional approaches to unsupervised representation learning, including variational autoencoders (VAEs), often struggle to capture the temporal and spectral diversity inherent in such data. Here we introduce variational decomposition autoencoding (VDA), a framework that extends VAEs by incorporating a strong structural bias toward signal decomposition. VDA is instantiated through variational decomposition autoencoders (DecVAEs), i.e., encoder-only neural networks that combine a signal decomposition model, a contrastive self-supervised task, and variational prior approximation to learn multiple latent subspaces aligned with time-frequency characteristics. We demonstrate the effectiveness of DecVAEs on simulated data and three publicly available scientific datasets, spanning speech recognition, dysarthria severity evaluation, and emotional speech classification. Our results demonstrate that DecVAEs surpass state-of-the-art VAE-based methods in terms of disentanglement quality, generalization across tasks, and the interpretability of latent encodings. These findings suggest that decomposition-aware architectures can serve as robust tools for extracting structured representations from dynamic signals, with potential applications in clinical diagnostics, human-computer interaction, and adaptive neurotechnologies.

Figures (6)

• Figure 1: $|$ Variational decomposition autoencoding (VDA) uses a neural network encoder to approximate latent distributions assuming a decomposition generative process and novel disentanglement mechanics.a, variational autoencoding directed factor graph and illustration of processes for prior approximation ($\phi$) and generation ($\theta$). b, (i) directed factor graph for the proposed VDA; (ii) VDA assumes that the prior distribution $p(z)$ is a composition of multiple frequency-related priors $p(z_1), p(z_2),..., p(z_C)$ and aims to recover these multiple priors with a single recognition model $q_\phi$. $q_\phi$ learns conditionally each subspace $z_c$ based on information from other subspaces $\{z_1,z_2,...,z_{C-1}\}$ and the observed input distributions $p_D(x)$ and its decomposed components $\{p_D(x_1),p_D(x_2),...,p_D(x_C)\}$, and approximates the true prior $p(z)$ with $q_\phi(z|x)$ based on aggregation of subspaces $z_c$. c, VDA requires additional structural biases to the Gaussian prior approximation, found in the decomposition reconstruction and orthogonality dynamics that promote a unique structure of the latent space that disentangles. DecVAE trains by optimizing the decomposition evidence lower bound, an extension of the classic evidence lower bound with a self-supervised loss that embodies the structural bias of decomposition (detailed in Methods). d, The variational decomposition autoencoder (DecVAE) can generalize disentanglement across several domains $\theta_n$ (speech, emotion, dysarthria severity). DecVAE can optionally recover generative processes in more than one time scales $K,M$ -where the $m$-th sequence contains $k$ frames- and apply a decomposition model $\mathcal{D}_w^C$ on input sequences $T$, and windowed frames $X$ from $T$, e.g. in the generative process of speech production ($\theta_1$) the long-term speaker identity variable $S$ influences short-term phonetic content $Z$. The decomposed time series $\{ t'^{(m)}, {t'}_1^{(m)}, ..., {t'}_C^{(m)}\}$, $\{ x'^{(k,m)}, {x'}_1^{(k,m)}, ..., {x'}_C^{(k,m)}\}$ at the two distinct time scales $T', X'$ are analyzed by different DecVAE branches $\phi_S,\phi_Z$ to learn latent representations on subspaces $\{ \tilde{s}_0^{(m)}, \tilde{s}_1^{(m)}, ..., \tilde{s}_C^{(m)}\}$, $\{ \tilde{z}_0^{(k,m)}, \tilde{z}_1^{(k,m)}, ..., \tilde{z}_C^{(k,m)}\}$. DecVAE approximates the true prior distributions $S$, $Z$ by disentangling inside the latent subspaces and then aggregating the subspaces through functions $h(\cdot), g(\cdot)$ to obtain $\tilde{S}, \tilde{Z}$ (detailed in Methods).
• Figure 2: $|$ Simulated speech data (SimVowels). a, TSNE maaten2008visualizing of development set frame-level vowel-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b, TSNE of training set sequence-level speaker-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the third component, after aggregation. c, TSNE of latent spaces using a as inputs (left) DecVAE with empirical wavelet transform (EWT) gilles2013ewt frequency-resonant frame embedding, (middle) DecVAE with EWT gilles2013ewt vowel-colored embedding, (right) VAE Kingma2014Auto-EncodingBayes vowel-colored embedding. d,TSNE of latent spaces using a,b as inputs, (left) DecVAE with EWT frequency-resonant sequence embedding, (mid-left) DecVAE with EWT gilles2013ewt speaker-colored sequence embedding, (mid-right) DecVAE with EWT gilles2013ewt speaker-colored frame embedding, (right) VAE Kingma2014Auto-EncodingBayes speaker-colored frame embedding. e, TSNE projection of DecVAE with EWT gilles2013ewt vowel-colored latent space (two batches) along with a multivariate Gaussian distribution (sphere). f, TSNE projection of VAE Kingma2014Auto-EncodingBayes vowel-colored latent space (two batches) along with a multivariate Gaussian distribution (sphere). g, (left) disentanglement (x-axis) versus informativeness (y-axis) from DCI Eastwood2018ARepresentations and robustness (circle size), (right) modularity (x-axis) versus explicitness (y-axis) Ridgeway2018LearningLoss and robustness suter2019irs(circle size). h, speaker identification (x-axis) versus phoneme (vowel) recognition (y-axis) and disentanglement Eastwood2018ARepresentations (circle size) performance of $\beta$-DecVAE variants and state-of-the-art methods VAE Kingma2014Auto-EncodingBayes, $\beta$-VAE Higgins2017Beta-VAE:Framework, ICA hyvarinen2001independent, PCA Greenacre2022PrincipalAnalysis ($\beta=0.1$).
• Figure 3: $|$ Real speech data TIMIT garofolo1993timit. a, TSNE maaten2008visualizing of a subset (13 phonemes from 6 batches) of the development set, frame-level phoneme-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b, TSNE of a subset (10 speakers from 6 batches) of the development set, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the third component, after aggregation. c, TSNE of latent spaces using a as inputs (6 batches) (left) DecVAE with EWT gilles2013ewt frequency-resonant frame embedding ($C=4$), (middle) DecVAE with EWT gilles2013ewt phoneme-colored embedding, (right) VAE Kingma2014Auto-EncodingBayes phoneme-colored embedding. d, TSNE of latent spaces using a,b as inputs (6 batches), (left) DecVAE with EWT gilles2013ewt frequency-resonant embedding, (mid-left) DecVAE with EWT gilles2013ewt speaker-colored embedding, (mid-right) DecVAE with EWT gilles2013ewt speaker-colored frame embedding, (right) VAE Kingma2014Auto-EncodingBayes speaker-colored frame embedding. e, TSNE projection of DecVAE with EWT gilles2013ewt phoneme-colored latent space (6 batches) along with a multivariate Gaussian distribution (sphere). f, TSNE projection of VAE Kingma2014Auto-EncodingBayes phoneme-colored latent space (6 batches) along with a multivariate Gaussian distribution (sphere). g, (left) disentanglement (x-axis) versus informativeness (y-axis) from DCI Eastwood2018ARepresentations and robustness suter2019irs (circle size), (right) modularity (x-axis) versus explicitness (y-axis) Ridgeway2018LearningLoss and robustness suter2019irs (circle size). h, total disentanglement performance of $\beta$-DecVAE variants and state-of-the-art methods VAE Kingma2014Auto-EncodingBayes, $\beta$-VAE Higgins2017Beta-VAE:Framework, ICA hyvarinen2001independent, PCA Greenacre2022PrincipalAnalysis.
• Figure 4: $|$ Zero-shot disentanglement on dysarthric speech data VOC-ALS Dubbioso2024VoiceControls. a, TSNE maaten2008visualizing of a subset (4 batches) of the dataset, frame-level King's clinical stage Roche2012ASclerosis-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b, TSNE of a subset (4 batches) of the dataset, frame-level phoneme-colored inputs (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the third component, after aggregation. c, TSNE of latent spaces using a as inputs and a pre-trained $\beta$-DecVAE on SimVowels ($\beta=0.1$) for zero-shot evaluation (4 batches) (left) $\beta$-DecVAE with FD, frequency-resonant frame embedding ($C=4$), (middle-left) $\beta$-DecVAE with FD, King's clinical stage Roche2012ASclerosis-colored embedding, (middle-right) $\beta$-DecVAE phoneme-colored embedding, (right) $\beta$-DecVAE speaker-colored embedding (for 10 speakers). d, TSNE of latent spaces using b as inputs and a pre-trained on SimVowels $\beta$-VAE ($\beta=0.1$) for zero-shot evaluation (4 batches), (left) $\beta$-VAE Higgins2017Beta-VAE:Framework King's clinical stage Roche2012ASclerosis-colored embedding, (mid-right) $\beta$-VAE Higgins2017Beta-VAE:Frameworkphoneme-colored embedding, (right) $\beta$-VAE Higgins2017Beta-VAE:Framework speaker-colored embedding (for 10 speakers). e, TSNE projection of $\beta$-DecVAE with FD, King's clinical stage Roche2012ASclerosis-colored latent space (4 batches) along with a multivariate Gaussian distribution (sphere). f, TSNE projection of $\beta$-VAE Higgins2017Beta-VAE:Framework King's clinical stage Roche2012ASclerosis-colored latent space (4 batches) along with a multivariate Gaussian distribution (sphere). g, (left) King's clinical stage Roche2012ASclerosis detection (x-axis) versus speaker identification (y-axis) and phoneme recognition (circle size), (right) disentanglement (x-axis) versus informativeness (y-axis) from DCI Eastwood2018ARepresentations and robustness suter2019irs (circle size). h, total classification performance (accuracies and F1-macro with $95\%$ confidence intervals error bars) of $\beta$-DecVAE variants and state-of-the-art methods VAE Kingma2014Auto-EncodingBayes, $\beta$-VAE Higgins2017Beta-VAE:Framework, ICA hyvarinen2001independent, PCA Greenacre2022PrincipalAnalysis, in predicting King's clinical stage, disease duration, phoneme recognition, speaker identification.
• Figure 5: $|$Fine-tuning disentanglement on emotional speech data IEMOCAP busso2008iemocap. a, TSNE maaten2008visualizing of a subset (6 batches) of the dataset, frame-level categorical emotion-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b, TSNE of a subset (6 batches) of the dataset, frame-level speaker-colored inputs (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the third component, after aggregation. c, TSNE of latent spaces using a as inputs and a pre-trained DecVAE on SimVowels for zero-shot evaluation (6 batches) (left) DecVAE with FD, frequency-resonant frame embedding ($C=4$), (middle-left) DecVAE with FD, emotion-colored embedding, (middle-right) $\beta$-DecVAE speaker-colored embedding, (right) $\beta$-DecVAE phoneme-colored embedding (for 11 phonemes). d, TSNE of latent spaces using b as inputs and a pre-trained on SimVowels $\beta$-VAE ($\beta=0.1$) for zero-shot evaluation (6 batches), (left) $\beta$-VAE Higgins2017Beta-VAE:Framework emotion-colored embedding, (mid-right) $\beta$-VAE Higgins2017Beta-VAE:Frameworkspeaker-colored embedding, (right) $\beta$-VAE Higgins2017Beta-VAE:Framework phoneme-colored embedding (for 11 phonemes). e, TSNE projection of DecVAE with FD, emotion-colored latent space (6 batches) along with a multivariate Gaussian distribution (sphere). f, TSNE projection of $\beta$-VAE Higgins2017Beta-VAE:Framework emotion-colored latent space (6 batches) along with a multivariate Gaussian distribution (sphere). g, (left) phoneme recognition (x-axis) versus speaker identification (y-axis) and emotion recognition (circle size) performances, (right) disentanglement (x-axis) versus informativeness (y-axis) from DCI Eastwood2018ARepresentations and robustness suter2019irs (circle size). h, total classification performance (weighted and unweighted accuracy, weighted F1 with $95\%$ confidence intervals error bars) of $\beta$-DecVAE variants and state-of-the-art methods VAE Kingma2014Auto-EncodingBayes, $\beta$-VAE Higgins2017Beta-VAE:Framework, ICA hyvarinen2001independent, PCA Greenacre2022PrincipalAnalysis, in predicting phoneme, speaker and categorical emotion classes.