Seeking the Sufficiency and Necessity Causal Features in Multimodal Representation Learning

TL;DR

This work conceptualizes multimodal representations as comprising modality-invariant and modality-specific components and formulates tractable optimization objectives that enable multimodal models to learn high-PNS representations.

Abstract

Probability of necessity and sufficiency (PNS) measures the likelihood of a feature set being both necessary and sufficient for predicting an outcome. It has proven effective in guiding representation learning for unimodal data, enhancing both predictive performance and model robustness. Despite these benefits, extending PNS to multimodal settings remains unexplored. This extension presents unique challenges, as the conditions for PNS estimation, exogeneity and monotonicity, need to be reconsidered in a multimodal context. We address these challenges by first conceptualizing multimodal representations as comprising modality-invariant and modality-specific components. We then analyze how to compute PNS for each component while ensuring non-trivial PNS estimation. Based on these analyses, we formulate tractable optimization objectives that enable multimodal models to learn high-PNS representations. Experiments demonstrate the effectiveness of our method on both synthetic and real-world data.

Figures (2)

• Figure 1: The causal graph showing data generation process with modality $M$
• Figure 2: A typical structure of a decomposition model and its adaptation to our method

Theorems & Definitions (5)

• Definition 1: PNS pearl2009causality
• Definition 2: Exogeneity pearl2009causality
• Definition 3: Monotonicity pearl2009causality
• Lemma 1: pearl2009causality
• Lemma 2: pearl2009causality