Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations

TL;DR

This work introduces the Multi-factor Sequential Disentanglement (MSD) benchmark, a modular framework for evaluating disentangled representations across six datasets spanning video, audio, and time series. It tackles the lack of standardized infrastructure, label scarcity, and labor-intensive evaluation by (i) standardizing datasets and metrics, (ii) proposing a post-hoc Latent Exploration Stage (LES) to auto-map latents to factors, (iii) introducing a Koopman-based Single Static Mode SKD (SSM-SKD) to disentangle static factors, and (iv) leveraging Vision-Language Models (VLMs) as taggers and zero-shot evaluators for automated annotation and assessment. Quantitative results show SSM-SKD achieving state-of-the-art on most benchmarks, while qualitative analyses and real-world failure cases reveal remaining gaps in capturing high-level semantics, especially in audio/time-series domains. The framework enables scalable, automated, cross-modal disentanglement research and provides a foundation for future improvements via more expressive generative models and integration with foundation models for richer, label-free evaluation.

Abstract

Learning disentangled representations in sequential data is a key goal in deep learning, with broad applications in vision, audio, and time series. While real-world data involves multiple interacting semantic factors over time, prior work has mostly focused on simpler two-factor static and dynamic settings, primarily because such settings make data collection easier, thereby overlooking the inherently multi-factor nature of real-world data. We introduce the first standardized benchmark for evaluating multi-factor sequential disentanglement across six diverse datasets spanning video, audio, and time series. Our benchmark includes modular tools for dataset integration, model development, and evaluation metrics tailored to multi-factor analysis. We additionally propose a post-hoc Latent Exploration Stage to automatically align latent dimensions with semantic factors, and introduce a Koopman-inspired model that achieves state-of-the-art results. Moreover, we show that Vision-Language Models can automate dataset annotation and serve as zero-shot disentanglement evaluators, removing the need for manual labels and human intervention. Together, these contributions provide a robust and scalable foundation for advancing multi-factor sequential disentanglement. Our code is available on GitHub, and the datasets and trained models are available on Hugging Face.

Figures (11)

• Figure 1: A visual summarization of our benchmark codebase and user flow. A full explanation of all the details of the flow can be found in App. \ref{['sec:codebase_details']}.
• Figure 2: Qualitative factor-swapping between two reference samples (left). For each row, a single factor from Sample 2 is swapped into Sample 1. Results are shown for two models: SSM-SKD (middle block) and MGP-VAE (right block). Factors include skin tone, pants color, shirt color, hair color, and movement dynamics (top to bottom).
• Figure 3: SSM-SKD results on VoxCelebOne: Row 1 shows the original samples (Orig.), row 2 the reconstructions (Rec.), and row 3 an attribute swap (Swap).
• Figure 4: Sampled frames from VoxCelebOne.
• Figure 5: Agreement between VLM-based and supervised evaluations.