An Information Criterion for Controlled Disentanglement of Multimodal Data

TL;DR

Multimodal data often entangles shared and modality-specific information, making $\text{MNI}$ unattainable in real settings. DisentangledSSL introduces a two-step, self-supervised optimization that learns a shared latent $Z_c$ and modality-specific latents $Z_s^1$, $Z_s^2$, guided by an information-theoretic objective and the IB curve to handle both attainable and unattainable MNI. The authors prove optimality guarantees for the learned shared representations and show that modality-specific representations achieve coverage and disentanglement under both regimes, with a tractable training objective combining InfoNCE and MI bounds. Empirically, DisentangledSSL improves downstream performance on vision-language prediction and molecule-phenotype retrieval across synthetic and real-world multimodal datasets, outperforming several baselines and demonstrating robust disentanglement.

Abstract

Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. We propose Disentangled Self-Supervised Learning (DisentangledSSL), a novel self-supervised approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly focusing on the scenario not covered in prior work where the so-called Minimum Necessary Information (MNI) point is not attainable. We demonstrate that DisentangledSSL successfully learns shared and modality-specific features on multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, as well as molecule-phenotype retrieval tasks for biological data. The code is available at https://github.com/uhlerlab/DisentangledSSL.

Figures (11)

• Figure 1: Post-perturbation phenotype ($X_1$) (i.e., cellular images or gene expression after the application of a drug to cells) and molecular structure ($X_2$) of an underlying drug perturbation system where cancer cells are targeted and killed while healthy cells remain unaffected. The Venn diagram illustrates shared and specific information between modalities $X_1$ and $X_2$: shared content is shown in red, modality-specific content in green, and entangled content due to unattainable MNI in orange. For example, for the drug mechanism, the molecular structure conveys full information, while the phenotype provides partial information (i.e. mechanisms causing cell death). Similarly, for the states of healthy cells, the phenotype specifies their cell states, whereas the molecular structure only indicates that the cells are unaffected without detailing their specific states.
• Figure 2: Graphical model.
• Figure 3: IB Curve
• Figure 4: Simulation study results.
• Figure 5: Results of different representations learned by DisentangledSSL.

Theorems & Definitions (21)

• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Proposition 1
• proof
• Proposition 2
• proof
• Proposition 3
• proof