Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainable Multimodal Regression via Information Decomposition

Zhaozhao Ma, Shujian Yu

TL;DR

The paper tackles the interpretability gap in multimodal regression by introducing PIDReg, a framework that embeds Partial Information Decomposition (PID) into end-to-end learning. It enforces Gaussianity in latent representations and uses a Gaussian PID with union information to obtain a tractable, closed-form decomposition into unique, redundant, and synergistic information, while CS-divergence regularizers promote Gaussianity and isolate modality-specific information. A two-stage optimization procedure learns modality encoders, fusion weights, and a predictor, with a linear-noise information bottleneck aiding generalization and interpretability. Empirical results across six real-world datasets, including large-scale brain age prediction, demonstrate improved predictive performance and clear modality-level explanations of contributions and interactions, enabling principled modality selection for efficient inference.

Abstract

Multimodal regression aims to predict a continuous target from heterogeneous input sources and typically relies on fusion strategies such as early or late fusion. However, existing methods lack principled tools to disentangle and quantify the individual contributions of each modality and their interactions, limiting the interpretability of multimodal fusion. We propose a novel multimodal regression framework grounded in Partial Information Decomposition (PID), which decomposes modality-specific representations into unique, redundant, and synergistic components. The basic PID framework is inherently underdetermined. To resolve this, we introduce inductive bias by enforcing Gaussianity in the joint distribution of latent representations and the transformed response variable (after inverse normal transformation), thereby enabling analytical computation of the PID terms. Additionally, we derive a closed-form conditional independence regularizer to promote the isolation of unique information within each modality. Experiments on six real-world datasets, including a case study on large-scale brain age prediction from multimodal neuroimaging data, demonstrate that our framework outperforms state-of-the-art methods in both predictive accuracy and interpretability, while also enabling informed modality selection for efficient inference. Implementation is available at https://github.com/zhaozhaoma/PIDReg.

Explainable Multimodal Regression via Information Decomposition

TL;DR

The paper tackles the interpretability gap in multimodal regression by introducing PIDReg, a framework that embeds Partial Information Decomposition (PID) into end-to-end learning. It enforces Gaussianity in latent representations and uses a Gaussian PID with union information to obtain a tractable, closed-form decomposition into unique, redundant, and synergistic information, while CS-divergence regularizers promote Gaussianity and isolate modality-specific information. A two-stage optimization procedure learns modality encoders, fusion weights, and a predictor, with a linear-noise information bottleneck aiding generalization and interpretability. Empirical results across six real-world datasets, including large-scale brain age prediction, demonstrate improved predictive performance and clear modality-level explanations of contributions and interactions, enabling principled modality selection for efficient inference.

Abstract

Multimodal regression aims to predict a continuous target from heterogeneous input sources and typically relies on fusion strategies such as early or late fusion. However, existing methods lack principled tools to disentangle and quantify the individual contributions of each modality and their interactions, limiting the interpretability of multimodal fusion. We propose a novel multimodal regression framework grounded in Partial Information Decomposition (PID), which decomposes modality-specific representations into unique, redundant, and synergistic components. The basic PID framework is inherently underdetermined. To resolve this, we introduce inductive bias by enforcing Gaussianity in the joint distribution of latent representations and the transformed response variable (after inverse normal transformation), thereby enabling analytical computation of the PID terms. Additionally, we derive a closed-form conditional independence regularizer to promote the isolation of unique information within each modality. Experiments on six real-world datasets, including a case study on large-scale brain age prediction from multimodal neuroimaging data, demonstrate that our framework outperforms state-of-the-art methods in both predictive accuracy and interpretability, while also enabling informed modality selection for efficient inference. Implementation is available at https://github.com/zhaozhaoma/PIDReg.
Paper Structure (67 sections, 1 theorem, 71 equations, 14 figures, 20 tables, 1 algorithm)

This paper contains 67 sections, 1 theorem, 71 equations, 14 figures, 20 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fusion Strategies in Multimodal Learning
  4. Interpretability in Multimodal Learning
  5. PIDReg: Partial Information Decomposition for Multimodal Regression
  6. Overall Framework
  7. Explainable Feature Fusion with Gaussian PID
  8. Gaussian Regularization of the Joint Distribution $P_{Z_1Z_2Y}$
  9. Regularizing for Unique Information Extraction
  10. Optimization and Algorithm
  11. Experiments
  12. Experiments on Synthetic Data
  13. Real-World Multimodal Regression
  14. Interpretability Analysis
  15. Conclusion
  16. ...and 52 more sections

Key Result

Proposition 1

Given $N$ observations $\{\mathbf{x}_{1,i},\mathbf{x}_{2,i},$$\mathbf{z}_{1,i}\}_{i=1}^N$ drawing from an unknown and fixed joint distribution $p(X_{1}, X_{2}, Z_{1})$ in which $\mathbf{x}_{1,i} \in \mathbb{R}^{d_1}$, $\mathbf{x}_{2,i}\in \mathbb{R}^{d_2}$, and $\mathbf{z}_{1,i} \in \mathbb{R}^{d}$.

Figures (14)

  • Figure 1: Framework of Partial Information Decomposition for Multimodal Regression (PIDReg), illustrated with video and audio modalities, where $P(X_{1})$, $P(X_{2})$, and $P(Y)$ denote empirical data distributions that may deviate from Gaussianity (e.g., skewed or heavy-tailed).
  • Figure 2: Estimated PID values when (a) $w_{u1}=0$, $w_{u2}=0$, $w_s=0.75$, $w_r=0.25$; (b) $w_{u1}=0$, $w_{u2}=0$, $w_s=0.50$, $w_r=0.50$; (c) $w_{u1}=0$, $w_{u2}=0$, $w_s=0.25$, $w_r=0.75$; (d) $w_{u1}=0$, $w_{u2}=0.80$, $w_s=0.10$, $w_r=0.10$; (e) $w_{u1}=0.80$, $w_{u2}=0$, $w_s=0.10$, $w_r=0.10$.
  • Figure 3: (a) Bias-corrected predicted age difference; (b, c) convergence curves of PID components.
  • Figure 4: (a) $D_{KL}(p;q)\to\infty$, $D_{KL}(q;p)\to\infty$, $D_{CS}(p;q)\to\infty$; (b) $D_{KL}(p;q)\to\infty$, $D_{KL}(q;p)\to\infty$, $D_{CS}(p;q)\ \text{finite}$; (c) $D_{KL}(p;q)\to\infty$, $D_{KL}(q;p)\to\infty$, $D_{CS}(p;q)\ \text{finite}$.
  • Figure 5: Distributions of $P(X_1)$, $P(X_2)$ and raw $P(Y)$.
  • ...and 9 more figures

Theorems & Definitions (2)

  • Definition 1: Union Information bertschinger2014quantifying
  • Proposition 1