Partial Information Decomposition for Continuous Variables based on Shared Exclusions: Analytical Formulation and Estimation

TL;DR

This work fills a gap in information-theoretic analysis by delivering a tractable analytic formulation and a practical estimator for a continuous Partial Information Decomposition based on shared exclusions, $I^{\mathrm{sx}}_\cap$. It derives a local, differentiable expression for redundancy and extends the KSG nearest-neighbor approach to handle disjunctive events, enabling multivariate PID with arbitrary numbers of sources. The authors validate the method on simple continuous toy gates and apply it to a simulated energy-management system, showing the method can reveal how environmental variables jointly inform a target quantity and how information is distributed among redundant, unique, and synergistic components. This framework, along with its multivariate generalization and estimator, provides a versatile tool for analyzing nonlinear dependencies in continuous systems across science and engineering, with practical applicability to real-world datasets and mixed-variable extensions discussed for future work.

Abstract

Describing statistical dependencies is foundational to empirical scientific research. For uncovering intricate and possibly non-linear dependencies between a single target variable and several source variables within a system, a principled and versatile framework can be found in the theory of Partial Information Decomposition (PID). Nevertheless, the majority of existing PID measures are restricted to categorical variables, while many systems of interest in science are continuous. In this paper, we present a novel analytic formulation for continuous redundancy--a generalization of mutual information--drawing inspiration from the concept of shared exclusions in probability space as in the discrete PID definition of $I^\mathrm{sx}_\cap$. Furthermore, we introduce a nearest-neighbor based estimator for continuous PID, and showcase its effectiveness by applying it to a simulated energy management system provided by the Honda Research Institute Europe GmbH. This work bridges the gap between the measure-theoretically postulated existence proofs for a continuous $I^\mathrm{sx}_\cap$ and its practical application to real-world scientific problems.

Figures (16)

• Figure 1: Partial Information Decomposition reveals the intricate interdependencies between classical information-theoretic quantities williams2010. Using PID, the mutual information $I(T:S_1, S_2)$ (large oval) and the marginal information terms $I(T:S_1)$ and $I(T:S_2)$ (circles) can simultaneously be dissected into a total of four information atoms (colored areas): The unique information that $S_1$ carries about $T$, $\Pi_\mathrm{unq,1}$ (orange), and likewise the unique information of $S_2$, $\Pi_\mathrm{unq,2}$ (red); the redundant information of $S_1$ and $S_2$ about $T$, $\Pi_\mathrm{red}$ (blue), and the synergistic information exclusively contained in $S_1$, and $S_2$, $\Pi_\mathrm{syn}$ (teal). The marginal mutual information terms $I(T:S_i)$ can be constructed from the redundant and the corresponding unique atoms, while the joint mutual information $I(T:S_1, S_2)$ contains of all four atoms.
• Figure 2: Local densities can be estimated from data in two opposite ways.A Kernel density estimation using a fixed volume $\mathrm{vol}(\epsilon)$ around the reference point $x_i$. B$k$-nearest-neighbor-based estimation using an adaptive search volume, determined by the distance to the $k$-th neighbor.
• Figure 3: The KSG estimator for mutual information kraskov works by considering $k$th nearest neighbors in the joint and marginal spaces in three steps.A Determining search radius in joint space B counting neighbors in marginal space of $S$ and C counting neighbors in marginal space of $T$.
• Figure 4: To estimate local redundant information for a two-variable example, the search regions of the original mutual information KSG estimator kraskov need to be adapted.A For mutual information estimation, the radius $\epsilon$ at a point $x$ is assessed in the joint space by searching for the minimal distance such that all of the variables' marginal distances to $x$ is smaller or equal to $\epsilon$. B For redundancy estimation, on the other hand, $\epsilon$ is inferred by searching for the minimal distance such that at least one of the variables' marginal distances to $x$ is smaller or equal to $\epsilon$.
• Figure 5: With increasing sample size $N$, the estimated PID atoms converge to the numerically evaluated analytical results. The subfigures show the four PID atoms $\Pi$ and the mutual information $I(T:\bm{S})$ for the continuous redundant, unique and copy gates as well as the sum example estimated using the nearest-neighbors approach outlined above with $k=4$. The colored bars to the right of the graphs correspond to the respective analytical value for each atom and gate from \ref{['tab:analytical_examples']}.

Theorems & Definitions (9)

• Definition 1
• Definition 2
• Definition 3
• Example 1
• Theorem 1
• proof
• Corollary 1
• proof
• Example 2