DiffSDA: Unsupervised Diffusion Sequential Disentanglement Across Modalities

TL;DR

DiffSDA tackles unsupervised sequential disentanglement across video, audio, and time-series by introducing a modal-agnostic diffusion framework that models interdependent static $s_0$ and dynamic $d_0^{1:V}$ latent factors. It advances a probabilistic formulation with latent diffusion and a single score-matching loss, integrated with a VQ-VAE backbone for high-resolution data. Empirical results across multiple modalities show state-of-the-art disentanglement performance, including a novel video-disentanglement evaluation protocol and zero-shot transfer capabilities. The work enables robust cross-modal representation learning with practical impact on downstream tasks and reinforces diffusion models as a unified tool for complex, real-world sequential data.

Abstract

Unsupervised representation learning, particularly sequential disentanglement, aims to separate static and dynamic factors of variation in data without relying on labels. This remains a challenging problem, as existing approaches based on variational autoencoders and generative adversarial networks often rely on multiple loss terms, complicating the optimization process. Furthermore, sequential disentanglement methods face challenges when applied to real-world data, and there is currently no established evaluation protocol for assessing their performance in such settings. Recently, diffusion models have emerged as state-of-the-art generative models, but no theoretical formalization exists for their application to sequential disentanglement. In this work, we introduce the Diffusion Sequential Disentanglement Autoencoder (DiffSDA), a novel, modal-agnostic framework effective across diverse real-world data modalities, including time series, video, and audio. DiffSDA leverages a new probabilistic modeling, latent diffusion, and efficient samplers, while incorporating a challenging evaluation protocol for rigorous testing. Our experiments on diverse real-world benchmarks demonstrate that DiffSDA outperforms recent state-of-the-art methods in sequential disentanglement.

Figures (47)

• Figure 1: DiffSDA processes sequences $\mathbf{x}_0^{1:V}$ via semantic and stochastic encoders (top and bottom). Their outputs $(\boldsymbol{s}_0, \boldsymbol{d}_0^{1:V}, \mathbf{x}_t^{1:V})$ are fed to a stochastic decoder yielding a denoised $\tilde{\mathbf{x}}_0^{1:V}$ (right).
• Figure 2: We present swap (left), zero-shot (middle), and multifactor disentanglement (right) results on multiple real-world and high-resolution visual datasets. See Sec. \ref{['sec:results']} for further details.
• Figure 3: We present dynamic swap results of our approach (third row) and SPYL (fourth row) on CelebV-HQ (left), VoxCeleb (middle), and TaiChi-HD (right).
• Figure 4: Zero-shot swap results, training on VoxCeleb and tested on CelebV-HQ or MUG.
• Figure 5: Rows A and B are two inputs from the test set. Row C shows a dynamic swap example, using the static of A and dynamics of B. In row D we extract the same person from A, but with the dynamics as labeled in B. Finally, in row E, we extract the same person from A with the dynamics that are predicted by the classifier.