Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning

TL;DR

This work addresses identifiability in nonlinear ICA for disentangled representation learning under practical constraints. It introduces a unified framework that combines sufficient changes via auxiliary variables with a sparse mixing procedure, supported by theoretical results that yield subspace- and component-wise identifiability. The authors instantiate the theory with CG-VAE and CG-GAN, incorporating a domain-encoding network and a sparse mixing constraint, and validate the approach on synthetic and real multi-domain image data, showing improved disentanglement and domain-aware controllability. The method promises more broadly applicable identifiability guarantees in real-world settings where domain coverage and full sparsity are challenging, enabling more robust and interpretable generative models across domains.

Abstract

Disentangled representation learning aims to uncover latent variables underlying the observed data, and generally speaking, rather strong assumptions are needed to ensure identifiability. Some approaches rely on sufficient changes on the distribution of latent variables indicated by auxiliary variables such as domain indices, but acquiring enough domains is often challenging. Alternative approaches exploit structural sparsity assumptions on the mixing procedure, but such constraints are usually (partially) violated in practice. Interestingly, we find that these two seemingly unrelated assumptions can actually complement each other to achieve identifiability. Specifically, when conditioned on auxiliary variables, the sparse mixing procedure assumption provides structural constraints on the mapping from estimated to true latent variables and hence compensates for potentially insufficient distribution changes. Building on this insight, we propose an identifiability theory with less restrictive constraints regarding distribution changes and the sparse mixing procedure, enhancing applicability to real-world scenarios. Additionally, we develop an estimation framework incorporating a domain encoding network and a sparse mixing constraint and provide two implementations based on variational autoencoders and generative adversarial networks, respectively. Experiment results on synthetic and real-world datasets support our theoretical results.

Figures (6)

• Figure 1: Example for Theorem 1 with ground-truth and estimated data generation processes. The red dashed lines denote the redundant estimated mixing edges.
• Figure 2: Illustrations of CG-VAE and CG-GAN, respectively. $\hat{\epsilon}$ and $\epsilon$ are the estimated and ground-truth noise variables, respectively. $\mathcal{F}_u$ denotes the domain-encoding neural networks. Note that in the CG-GAN, we partition the latent variables into the domain-invariant content variables ${\mathbf{z}}_c$ and domain-specific style ${\mathbf{z}}_s$. The $L_m$ denotes the mask restriction for automatically optimal dimension determination. And $Y$ and $\hat{Y}$ denote the ground-truth and predicted labels of real and fake samples.
• Figure 3: Experiments results on synthetic datasets. The horizontal axis represents the number of auxiliary variables, and the vertical axis represents the values of MCC.
• Figure 4: Samples of multi-domain image generation on the CelebA and MNIST datasets. i-StyleGAN, CG-VAE, and CG-GAN share the same noise input $\epsilon$. We find that when the number domain is insufficient, iStyleGAN will produce unnecessary changes.
• Figure 5: The t-SNE visualization of different methods. Blue and red points represent features from generated images, while green and yellow points correspond to features from real images. A greater overlap between generated and real points within the same domain reflects improved performance.

Theorems & Definitions (11)

• Definition 1: Subspace-wise Identifiability of Latent Variables li2024subspace
• Definition 2: Component-wise Identifiability of Latent Variables hyvarinen2016unsupervised
• Theorem 1
• Theorem 2
• Corollary 1
• Theorem 1: Subspace Identification with Complementary Gains
• proof
• Theorem 2
• proof
• Corollary 1