Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions
Wenyuan Zhao, Adithya Balachandran, Chao Tian, Paul Pu Liang
TL;DR
The paper tackles efficient, accurate PID estimation for continuous multimodal data by proving that GPID optimally uses joint Gaussian marginals and by introducing Thin-PID, a gradient-based solver leveraging a Gaussian broadcast-channel view. To extend beyond Gaussianity, it proposes Flow-PID, which learns information-preserving latent Gaussian encoders via Cartesian normalizing flows and a Gaussian marginal loss, preserving total MI under bijections. Empirical results on canonical Gaussian and non-Gaussian synthetic data, along with real-world benchmarks (e.g., MultiBench, TCGA-BRCA, VQA), show Flow-PID provides accurate, scalable PID estimates and improves modality-interaction interpretability and model selection. The work offers a practical framework for quantifying redundant, unique, and synergistic information across modalities, enabling robust decision-making in multimodal modeling.
Abstract
The study of multimodality has garnered significant interest in fields where the analysis of interactions among multiple information sources can enhance predictive modeling, data fusion, and interpretability. Partial information decomposition (PID) has emerged as a useful information-theoretic framework to quantify the degree to which individual modalities independently, redundantly, or synergistically convey information about a target variable. However, existing PID methods depend on optimizing over a joint distribution constrained by estimated pairwise probability distributions, which are costly and inaccurate for continuous and high-dimensional modalities. Our first key insight is that the problem can be solved efficiently when the pairwise distributions are multivariate Gaussians, and we refer to this problem as Gaussian PID (GPID). We propose a new gradient-based algorithm that substantially improves the computational efficiency of GPID based on an alternative formulation of the underlying optimization problem. To generalize the applicability to non-Gaussian data, we learn information-preserving encoders to transform random variables of arbitrary input distributions into pairwise Gaussian random variables. Along the way, we resolved an open problem regarding the optimality of joint Gaussian solutions for GPID. Empirical validation in diverse synthetic examples demonstrates that our proposed method provides more accurate and efficient PID estimates than existing baselines. We further evaluate a series of large-scale multimodal benchmarks to show its utility in real-world applications of quantifying PID in multimodal datasets and selecting high-performing models.