Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling

TL;DR

PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting, and visually demonstrates how varying completions of the missing modality result in a set of plausible labels.

Abstract

Despite the recent success of Multimodal Large Language Models (MLLMs), existing approaches predominantly assume the availability of multiple modalities during training and inference. In practice, multimodal data is often incomplete because modalities may be missing, collected asynchronously, or available only for a subset of examples. In this work, we propose PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting. PRIMO enables the use of all available training examples, whether modalities are complete or partial. Specifically, it models the missing modality through a latent variable that captures its relationship with the observed modality in the context of prediction. During inference, we draw many samples from the learned distribution over the missing modality to both obtain the marginal predictive distribution (for the purpose of prediction) and analyze the impact of the missing modalities on the prediction for each instance. We evaluate PRIMO on a synthetic XOR dataset, Audio-Vision MNIST, and MIMIC-III for mortality and ICD-9 prediction. Across all datasets, PRIMO obtains performance comparable to unimodal baselines when a modality is fully missing and to multimodal baselines when all modalities are available. PRIMO quantifies the predictive impact of a modality at the instance level using a variance-based metric computed from predictions across latent completions. We visually demonstrate how varying completions of the missing modality result in a set of plausible labels.

Figures (15)

• Figure 1: Overview of PRIMO. Given an observed modality $\mathbf{x}_{\text{o}}$ and an additional modality $\mathbf{x}_{\text{m}}$ that may be missing, PRIMO samples a latent variable $\mathbf{z}$ conditioned on the available modalities. The classifier maps $(\mathbf{x}_{\text{o}},\mathbf{z})$ to predictions, and the conditional variance $\mathcal{V}_{\mathbf{z}}\left[p(\mathbf{y} \mid \mathbf{x}_{\text{o}},\mathbf{z})\right]$ quantifies how changes in $\mathbf{z}$ affect the prediction. When both modalities are observed (orange), $\mathcal{V}$ is lower. When a modality is missing (red), $\mathcal{V}$ is higher. PRIMO then clusters the output logits across latent samples to visualize plausible labels under each availability scenario.
• Figure 1: Accuracy on AV-MNIST. We consider audio-missing and vision-missing settings. PRIMO performs comparably to the unimodal baseline that uses the available modality and to the multimodal baseline in both scenarios.
• Figure 2: DGP for missing modalities. The dashed line denotes an a priori correlation between the two modalities.
• Figure 2: MIMIC-III accuracy. We consider mortality and ICD-9 group prediction under missing and complete modality settings. We report mean and standard deviation across five runs for the unimodal baseline ($\mathbf{x}_{\text{o}}$), the multimodal baseline $(\mathbf{x}_{\text{o}},\mathbf{x}_{\text{m}})$, and PRIMO in each setting.
• Figure 3: Evaluation on the XOR dataset.(Left.) Accuracy under complete and missing-modality inputs. PRIMO matches the unimodal baseline $(\mathbf{x}_{\text{o}})$ when $\mathbf{x}_{\text{m}}$ is missing and matches the multimodal baseline $(\mathbf{x}_{\text{o}},\mathbf{x}_{\text{m}})$ when both modalities are observed, outperforming the remaining baselines. (Right.) Scatter plot of the predictive impact gap $\mathcal{V}_{\text{missing}}-\mathcal{V}_{\text{complete}}$. The gap is small for examples with $\mathbf{x}_{\text{o}}>0$, where the label can be determined by $\mathbf{x}_{\text{o}}$ only, and larger for $\mathbf{x}_{\text{o}}<0$, where $\mathbf{x}_{\text{m}}$ affects the label.