Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints

TL;DR

V3 introduces a domain-general, unsupervised method for disentangling content and style via variance-versus-invariance priors applied to a vector-quantized autoencoder. It defines variability statistics and four hinge-based regularizers to enforce distinct within- and across-sample patterns for content and style, achieving robust cross-domain disentanglement across audio, image, and video datasets. The approach yields not only stronger content-style separation than unsupervised baselines but also superior few-shot OOD generalization and interpretable, near one-to-one symbolic mappings in the content codebook. These results suggest practical benefits for controllable generation, style transfer, and symbolic reasoning, with potential applicability to more complex, unsegmented data in the future.

Abstract

We contribute an unsupervised method that effectively learns disentangled content and style representations from sequences of observations. Unlike most disentanglement algorithms that rely on domain-specific labels or knowledge, our method is based on the insight of domain-general statistical differences between content and style -- content varies more among different fragments within a sample but maintains an invariant vocabulary across data samples, whereas style remains relatively invariant within a sample but exhibits more significant variation across different samples. We integrate such inductive bias into an encoder-decoder architecture and name our method after V3 (variance-versus-invariance). Experimental results show that V3 generalizes across multiple domains and modalities, successfully learning disentangled content and style representations, such as pitch and timbre from music audio, digit and color from images of hand-written digits, and action and character appearance from simple animations. V3 demonstrates strong disentanglement performance compared to existing unsupervised methods, along with superior out-of-distribution generalization under few-shot adaptation compared to supervised counterparts. Lastly, symbolic-level interpretability emerges in the learned content codebook, forging a near one-to-one alignment between machine representation and human knowledge.

Figures (18)

• Figure 1: An illustration of the variance-versus-invariance constraints in content and style. Here, content refers to the symbols. Each row represents a data sample, which is divided into multiple fragments along columns. Each fragment contains one content-style pair. For example, digits (e.g., 9, 8, 6) can represent content, while colors (e.g., orange, brown, teal) represent style.
• Figure 2: The model architecture of V3. Left: The autoencoder has two branches for content and style respectively, where the content branch has a VQ layer at the encoder output. Right: the V3 constraints, where double-dashed arrows represent measuring the variability by $\nu_{k}(\cdot)$, and solid arrows represent taking the average.
• Figure 3: Comparison of generated images by recombining ${\bm{z}}^{\mathrm{s}}$ from given sources in SVHN and all ${\bm{z}}^{\mathrm{c}}$ in the learned codebook.
• Figure 4: Left: example training data in PhoneNums. Right: example data for out-of-distribution evaluation in PhoneNums.
• Figure 5: Example data in the original SVHN dataset. The digits in the images are bounded by the red boxes.