Disentangled Representation Learning via Modular Compositional Bias

TL;DR

This work tackles the challenge of disentangled representation learning by removing factor-specific biases from architectures and learning objectives and replacing them with a modular mixing strategy. By randomly remixing latent factors from two images and enforcing realism via a diffusion prior and alignment via a compositional consistency loss, the method can disentangle attributes, objects, or both without changing network architectures. The approach yields state-of-the-art results on attribute disentanglement, competitive object disentanglement, and uniquely achieves joint disentanglement of global style and objects on a new CLEVR-Style dataset. Overall, the modular compositional bias offers a scalable, general framework for DRL with broad applicability and easier adaptation to new variation factors. The math centers on maximizing the composite data likelihood $p( ilde{x})$ alongside a compositional consistency objective, formalized through $\mathcal{L}_{\text{Prior}}$ and $\mathcal{L}_{\text{Con}}$, enabling flexible factor disentanglement via mixing strategies $\pi_{attr}$, $\pi_{obj}$, and $\pi_{joint}$.

Abstract

Recent disentangled representation learning (DRL) methods heavily rely on factor specific strategies-either learning objectives for attributes or model architectures for objects-to embed inductive biases. Such divergent approaches result in significant overhead when novel factors of variation do not align with prior assumptions, such as statistical independence or spatial exclusivity, or when multiple factors coexist, as practitioners must redesign architectures or objectives. To address this, we propose a compositional bias, a modular inductive bias decoupled from both objectives and architectures. Our key insight is that different factors obey distinct recombination rules in the data distribution: global attributes are mutually exclusive, e.g., a face has one nose, while objects share a common support (any subset of objects can co-exist). We therefore randomly remix latents according to factor-specific rules, i.e., a mixing strategy, and force the encoder to discover whichever factor structure the mixing strategy reflects through two complementary objectives: (i) a prior loss that ensures every remix decodes into a realistic image, and (ii) the compositional consistency loss introduced by Wiedemer et al. (arXiv:2310.05327), which aligns each composite image with its corresponding composite latent. Under this general framework, simply adjusting the mixing strategy enables disentanglement of attributes, objects, and even both, without modifying the objectives or architectures. Extensive experiments demonstrate that our method shows competitive performance in both attribute and object disentanglement, and uniquely achieves joint disentanglement of global style and objects. Code is available at https://github.com/whieya/Compositional-DRL.

Figures (23)

• Figure 1: Overview of our method. To derive a compositional bias, we analyze the compositional structure of attribute and object, and implement it as a mixing strategy. Given a mixing strategy, we decode composite representations into an image and minimize our prior loss and consistency loss to ensure it is both realistic and aligned with the latents. Note that the figure illustrates a specific example for object mixing strategy.
• Figure 1: Quantitative results on attribute disentanglement. Our method achieves state-of-the-art performance in almost all of the datasets, except FactorVAE score in Cars3D. We report mean of 10-repeated runs. Standard deviations are in Appendix \ref{['subsec:random_seeds']}.
• Figure 2: Comparisons on object property prediction. Ours achieves performance comparable to state-of-the-art methods. For the position$^*$ of CLEVREasy, we use the discrete labels in the dataset and reports the accuracy.
• Figure 4: Object segmentation results on CLEVRTex. Despite lacking a built-in spatial clustering mechanism (e.g., slot-attention), our method combined with a Spatial Broadcast Decoder consistently captures complete objects, whereas the baselines often split objects across multiple latents.
• Figure 4: Evaluation of joint disentanglement on CLEVR-Style dataset.