Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures

TL;DR

This work critically analyzes Partial Information Decomposition (PID) for multivariate systems, showing that the standard lattice-based approach cannot maintain subsystem consistency when three or more sources are involved, due to high-order synergy. It delivers a closed-form two-source PID that satisfies all axioms via a max-entropy, pairwise-preserving construction using an auxiliary variable S1'. To address the multivariate inconsistency, the authors propose a lattice-free framework defining multivariate unique information and synergy (Un and Syn) along with higher-order measures SE_K and Total Synergistic Effect (TSE), validated through Ising-model experiments where redundancy and unique information peak near criticality while synergy signals emergent order. The results motivate a shift away from lattice-based PID toward robust multivariate information measures and new theoretical foundations for high-order interactions in complex systems. Together, the contributions advance both the theory and application of information decomposition in physics, neuroscience, and related fields, while underscoring the need for a new structural paradigm for multivariate information.

Abstract

While mutual information effectively quantifies dependence between two variables, it does not by itself reveal the complex, fine-grained interactions among variables, i.e., how multiple sources contribute redundantly, uniquely, or synergistically to a target in multivariate settings. The Partial Information Decomposition (PID) framework was introduced to address this by decomposing the mutual information between a set of source variables and a target variable into fine-grained information atoms such as redundant, unique, and synergistic components. In this work, we review the axiomatic system and desired properties of the PID framework and make three main contributions. First, we resolve the two-source PID case by providing explicit closed-form formulas for all information atoms that satisfy the full set of axioms and desirable properties. Second, we prove that for three or more sources, PID suffers from fundamental inconsistencies: we review the known three-variable counterexample where the sum of atoms exceeds the total information, and extend it to a comprehensive impossibility theorem showing that no lattice-based decomposition can be consistent for all subsets when the number of sources exceeds three. Finally, we deviate from the PID lattice approach to avoid its inconsistencies, and present explicit measures of multivariate unique and synergistic information. Our proposed measures, which rely on new systems of random variables that eliminate higher-order dependencies, satisfy key axioms such as additivity and continuity, provide a robust theoretical explanation of high-order relations, and show strong numerical performance in comprehensive experiments on the Ising model. Our findings highlight the need for a new framework for studying multivariate information decomposition.

Figures (15)

• Figure 1: A pictorial representation of Partial Information Decomposition \ref{["equ:Information Atoms' relationship_1"]}, where $I((S_1,S_2);T)$ is decomposed to its finer information atoms, the synergistic $\operatorname{Syn}(S_1,S_2\to T)$ (also called "complementary"), the redundant $\operatorname{Red}(S_1,S_2\to T)$ (also called "shared"), and the two directional unique components $\operatorname{Un}(S_1\to T|S_2)$ and $\operatorname{Un}(S_2\to T|S_1)$. The summation of the redundant atom and one of the unique atoms must be equal to the corresponding mutual information, as described in Eq. \ref{["equ:Information Atoms' relationship_2"]}.
• Figure 2: The structure of PID with 3 source variables.
• Figure 3: Construction of the two systems $(\hat{S}_1,\hat{S}_2,\hat{S}_3,\hat{T})$ and $(\bar{S}_1,\bar{S}_2,\bar{S}_3,\bar{T})$ used in Lemma \ref{['lemma:NoUniversalSubset']}. Although the two systems induce identical PI-atoms under any lattice-based decomposition satisfying Axioms \ref{['axiom: commutativity']}--\ref{['axiom: Self-redundancy']}, they differ in total mutual information, illustrating the impossibility of a universal reconstruction map from atoms to $I(\mathbf{S};T)$, as formalized in Theorem \ref{['theorem:no sub set']}.
• Figure : (A)
• Figure : (A)

Theorems & Definitions (33)

• Definition 1: PID Redundancy Lattice
• Definition 2: Partial Information Decomposition Framework
• Lemma 1: Subsystem Consistency
• Corollary 1
• Lemma 2: Nonnegativity
• proof
• Remark 1
• Definition 3: Unique Information in two source variable systems
• Lemma 3: Redundant Information in two source variable systems
• Lemma 4: Synergistic Information in two source variable systems