Cross-Modal Alignment via Variational Copula Modelling

TL;DR

The paper addresses the challenge of aligning and fusing heterogeneous modalities by modeling the joint distribution via a copula-based framework. It introduces CM$^{2}$, which uses Gaussian mixture marginals for each modality and a copula to capture cross-modal dependencies, enabling robust handling of missing modalities through sampling from learned marginals. The approach is optimized with stochastic variational inference and grounded by Sklar's theorem, guaranteeing a valid joint distribution given the marginals. Empirical results on MIMIC datasets show consistent improvements over state-of-the-art baselines in both fully observed and partially observed modality scenarios, with ablations confirming the importance of copula-based alignment and fusion components. The work demonstrates that copula-driven multimodal learning can yield more accurate representations and robust imputations, offering a principled alternative to simple fusion strategies and highlighting potential extensions to other domains requiring distribution alignment and multi-feature integration.

Abstract

Various data modalities are common in real-world applications (e.g., electronic health records, medical images and clinical notes in healthcare). It is essential to develop multimodal learning methods to aggregate various information from multiple modalities. The main challenge is how to appropriately align and fuse the representations of different modalities into a joint distribution. Existing methods mainly rely on concatenation or the Kronecker product, oversimplifying the interaction structure between modalities and indicating a need to model more complex interactions. Additionally, the joint distribution of latent representations with higher-order interactions is underexplored. Copula is a powerful statistical structure for modelling the interactions among variables, as it naturally bridges the joint distribution and marginal distributions of multiple variables. We propose a novel copula-driven multimodal learning framework, which focuses on learning the joint distribution of various modalities to capture the complex interactions among them. The key idea is to interpret the copula model as a tool to align the marginal distributions of the modalities efficiently. By assuming a Gaussian mixture distribution for each modality and a copula model on the joint distribution, our model can generate accurate representations for missing modalities. Extensive experiments on public MIMIC datasets demonstrate the superior performance of our model over other competitors. The code is available at https://github.com/HKU-MedAI/CMCM.

Figures (5)

• Figure 1: Overview of our proposed CM$^{2}$ framework. For a dataset with $M$ modalities, we extract modality-specific embeddings $\bm{z}_m$ via $\text{Encoder}_m$ and compute its Gaussian mixture model (GMM). We then model the marginal distributions and estimate the joint distribution using a copula family $C$. We sample $\bm{\hat{z}}_m$ from its GMM if modality $m$ is missing. The concatenated embedding $\bm{z}$ then passes through a 2-layer LSTM fusion module and MLP classifier to predict $\bm{\hat{y}}$. The ELBO for backpropagation can be obtained by aggregating the task-specific loss (e.g., cross-entropy loss) and the negative log-likelihood from the joint distribution.
• Figure 2: Plots of the fitted copula density to demonstrate the interrelationship captured by the copula model (Left: Gumbel, middle: Gaussian, right: Frank).
• Figure 3: Plots comparing the value of $\alpha$ and the correlation,$\text{Corr} = {(\alpha-1)}/{\alpha}$ learned by the Gumbel copula model.
• Figure 4: Plots of the copula densities of the Gumbel family at epochs 5, 50, and 100, respectively.
• Figure 5: Results (left: AUROC; right: AUPR) of CM$^{2}$ on MIMIC-IV, where the model reduces to a multivariate Gaussian disdtribution when $K=1$.

Theorems & Definitions (1)

• Theorem 3.1