Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals

Mathew Chandy, Michael Johnson, Judong Shen, Devan V. Mehrotra, Hua Zhou, Jin Zhou, Xiaowu Dai

TL;DR

This work addresses the challenge of identifying informative data modalities in multimodal learning while quantifying uncertainty in their contributions. It introduces conformal Shapley intervals, which fuse Shapley attribution with conditional conformal inference to yield input-dependent uncertainty intervals for modality importance. A modality-selection procedure with provable near-optimality guarantees is built on these intervals, enabling adaptive, covariate-aware model reduction. Empirical results on synthetic data, MNIST, and ADNI demonstrate valid uncertainty quantification and strong predictive performance using relatively few modalities, with insights that vary across subgroups. The approach advances interpretability, reliability, and efficiency in multimodal learning by unifying attribution, uncertainty, and decision-making in a single framework.

Abstract

Multimodal learning combines information from multiple data modalities to improve predictive performance. However, modalities often contribute unequally and in a data dependent way, making it unclear which data modalities are genuinely informative and to what extent their contributions can be trusted. Quantifying modality level importance together with uncertainty is therefore central to interpretable and reliable multimodal learning. We introduce conformal Shapley intervals, a framework that combines Shapley values with conformal inference to construct uncertainty-aware importance intervals for each modality. Building on these intervals, we propose a modality selection procedure with a provable optimality guarantee: conditional on the observed features, the selected subset of modalities achieves performance close to that of the optimal subset. We demonstrate the effectiveness of our approach on multiple datasets, showing that it provides meaningful uncertainty quantification and strong predictive performance while relying on only a small number of informative modalities.

Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals

TL;DR

This work addresses the challenge of identifying informative data modalities in multimodal learning while quantifying uncertainty in their contributions. It introduces conformal Shapley intervals, which fuse Shapley attribution with conditional conformal inference to yield input-dependent uncertainty intervals for modality importance. A modality-selection procedure with provable near-optimality guarantees is built on these intervals, enabling adaptive, covariate-aware model reduction. Empirical results on synthetic data, MNIST, and ADNI demonstrate valid uncertainty quantification and strong predictive performance using relatively few modalities, with insights that vary across subgroups. The approach advances interpretability, reliability, and efficiency in multimodal learning by unifying attribution, uncertainty, and decision-making in a single framework.

Abstract

Multimodal learning combines information from multiple data modalities to improve predictive performance. However, modalities often contribute unequally and in a data dependent way, making it unclear which data modalities are genuinely informative and to what extent their contributions can be trusted. Quantifying modality level importance together with uncertainty is therefore central to interpretable and reliable multimodal learning. We introduce conformal Shapley intervals, a framework that combines Shapley values with conformal inference to construct uncertainty-aware importance intervals for each modality. Building on these intervals, we propose a modality selection procedure with a provable optimality guarantee: conditional on the observed features, the selected subset of modalities achieves performance close to that of the optimal subset. We demonstrate the effectiveness of our approach on multiple datasets, showing that it provides meaningful uncertainty quantification and strong predictive performance while relying on only a small number of informative modalities.
Paper Structure (25 sections, 5 theorems, 45 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 25 sections, 5 theorems, 45 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Traditional Shapley Values
  4. Multimodal Learning
  5. Conformal Inference
  6. Conformal Shapley Intervals
  7. Prediction-Based Value Functions
  8. Construction of Conformal Shapley Intervals
  9. Modality Selection and Inference via Conformal Shapley Intervals
  10. Theoretical Guarantees
  11. Empirical Results
  12. Synthetic Regression
  13. Results.
  14. MNIST Classification
  15. Results.
  16. ...and 10 more sections

Key Result

Lemma 1

Let $\mathcal{H}$ be any vector space, and assume that for all $f, g \in \mathcal{H}$, the partial derivative of $\mathcal{R}(g + \epsilon f)$ with respect to $\epsilon$ exists. If $f$ returns nonnegative values with $\mathbb E_P[f(X)] > 0$, then the prediction set given by $\hat{C}_j(x^{n+1})$ sati If we suppose that $\{(X^i, Y^i)\}_{i=1}^{n+1} \overset{i.i.d.}\sim P$, then for $f \in \mathcal{F}

Figures (4)

  • Figure 1: Diagram of conformal Shapley intervals in multimodal learning.
  • Figure 2: Model selection paths for synthetic regression data set, MNIST, and ADNI. Test MSE and $R^2$ are shown for regression and ADNI data, whereas test cross-entropy (CE) and accuracy (%) shown for MNIST.
  • Figure 3: Left panel: Mean Shapley values $\hat{\varphi}_j$ for each image patch $j$, averaged over the calibration set. Right panel: Selection frequency of each patch when at most one patch is selected ($q=1$).
  • Figure 4: Marginal distributions of Shapley values by modality in the ADNI experiment.

Theorems & Definitions (5)

  • Lemma 1: Conditional Coverage of Conformal Shapley Intervals
  • Theorem 1: Near-Optimal Utility of Selected Set for Classification
  • Theorem 2: Near-Optimal Utility of Selected Set for Regression
  • Lemma 2: Interpolation Error Bound
  • Lemma 3: RKHS Width Bound