Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning
Yuhang Liu, Zhen Zhang, Dong Gong, Erdun Gao, Biwei Huang, Mingming Gong, Anton van den Hengel, Kun Zhang, Javen Qinfeng Shi
TL;DR
This work challenges the necessity of DAGs in multimodal causal modeling by introducing a latent partial causal framework with two coupled latent variables $(oldsymbol{z}_x,oldsymbol{z}_t)$ linked by an undirected edge. It shows, under specific statistical assumptions, that representations learned by multimodal contrastive learning recover these latent variables up to structured transformations: orthogonal linearity on a hypersphere and permutation on a convex body, implying component-wise disentanglement. The authors connect the generative model to the inference objective, provide identifiability guarantees in two geometric settings, and propose practical pipelines (e.g., PCA+FastICA) to operationalize disentangled CLIP-like representations. Synthetic experiments establish robustness to assumption violations, while real-world CLIP-based evaluations demonstrate disentangled representations that enable few-shot learning and domain generalization across diverse datasets. Overall, the paper broadens the utility of multimodal contrastive learning by relaxing DAG constraints and offering principled, transferable disentanglement insights applicable to pre-trained models like CLIP.
Abstract
Directed acyclic graphs (DAGs) are fundamental graph structures in causal modeling, but identifying the desired DAG from observational data often requires strong assumptions that may not hold in real-world scenarios, especially for latent causal models and complex multimodal data. This raises the question of whether we can relax or bypass the DAG assumption while maintaining practical utility. In this work, we propose a novel latent partial causal model for multimodal data, featuring two latent coupled variables, connected by an undirected edge, to represent the transfer of knowledge across modalities. Under specific statistical assumptions, we establish an identifiability result, demonstrating that representations learned by multimodal contrastive learning correspond to the latent coupled variables up to a trivial transformation. This result deepens our understanding of the why multimodal contrastive learning works, highlights its potential for disentanglement, and expands the utility of pre-trained models like CLIP. Synthetic experiments confirm the robustness of our findings, even when the assumptions are partially violated. Most importantly, experiments on a pre-trained CLIP model embodies disentangled representations, enabling few-shot learning and improving domain generalization across diverse real-world datasets. Together, these contributions push the boundaries of multimodal contrastive learning, both theoretically and, crucially, in practical applications.