The problem of infinite information flow

TL;DR

The paper investigates conditional mutual information and its network generalizations in a dynamic setting where $X=T(Y,Z)$, revealing a zero-infinity dichotomy: TE is either $0$ or $+\infty$ under mild continuity, which hinders comparing information flow magnitudes. To address this, the authors introduce a discretization framework and a conjectured relative-ambiguity formula $A_T(Y)$, showing that discretized MI satisfies $I(X^{\Delta};Y^{\Delta})+\ln\Delta\to -A_T(Y)$ as $\Delta\to0^+$, thereby producing finite, comparable measures that reflect the system's ambiguity. They establish a disintegration-based expression for conditional mutual information, $I(X;Y|Z)=\int I(X_z;Y_z)\,dP_Z(z)$, reducing the analysis to conditioned MI on $Z$-slices. Numerical experiments on Bernoulli interval maps and sine box functions validate the discretization approach and illustrate how relative ambiguity tracks changes in information flow, providing a practical path to distinguishing flow magnitudes in deterministic dynamical systems.

Abstract

We study conditional mutual information (cMI) between a pair of variables $X,Y$ given a third one $Z$ and derived quantities including transfer entropy (TE) and causation entropy (CE) in the dynamically relevant context where $X=T(Y,Z)$ is determined by $Y,Z$ via a deterministic transformation $T$. Under mild continuity assumptions on their distributions, we prove a zero-infinity dichotomy for cMI for a wide class of $T$, which gives a yes-or-no answer to the question of information flow as quantified by TE or CE. Such an answer fails to distinguish between the relative amounts of information flow. To resolve this problem, we propose a discretization strategy and a conjectured formula to discern the \textit{relative ambiguities} of the system, which can serve as a reliable proxy for the relative amounts of information flow. We illustrate and validate this approach with numerical evidence.

Figures (3)

• Figure 1: Disintegrated distributions. The main histogram at the bottom illustrates the distribution $P_Z$ of variable $Z$, which, together with $Y$, determines $X=T(Y,Z)$ via a measurable map $T$. The joint distribution $P_{XYZ}$ disintegrates into $(P_{XYZ})_z$ for each realization of $Z=z$, which can be interpreted as the joint distribution $P_{X_z Y_z}$ of the conditioned versions $X_z,Y_z$ of $X,Y$. The left, center and right subplots above the main histogram illustrate three typical disintegrated distributions $(P_{XYZ})_z = P_{X_z Y_z}$, where $X_z$ follows a constant, atomic and continuous distribution, respectively. In each subplot, the scatter plot shows the joint distribution $P_{X_zY_z}$, the top histogram shows the marginal distribution $P_{Y_z}$, and the right histogram shows the marginal distribution $P_{X_z}$. The intensity of the blue gradient indicates regions of high probability density.
• Figure 2: Discretization via uniform $\Delta^{-1}=300$ partition of continuous random variable $X=E_d(Y)$ determined by variable $Y$ via the Bernoulli map $E_d:x\mapsto d\cdot x\mod 1$. In the left and middle panels, the scatter plots show the joint distribution $P_{X^{\Delta}Y^{\Delta}}$ of the discretized variables $X^{\Delta},Y^{\Delta}$, together with the marginal distributions $P_{Y^{\Delta}}$ at the top and $P_{X^{\Delta}}$ on the right. The blue and red plots correspond to $Y$ following the uniform and Gaussian $\mathcal{N}_{[0,1]}(0.3,0.02)$ distributions, respectively. Here, $\mathcal{N}_{[0,1]}(0.3,0.02)$ means the Gaussian distribution centered at $0.3$ with variance $0.02$ and truncated between 0 and 1. The right panel plots for each Bernoulli expansion rate $d$, the corresponding $I(X^{\Delta};Y^{\Delta})$ of the discretized variables. The blue and red dots correspond to the empirical calculations of uniform and Gaussian $\mathcal{N}_{[0,1]}(0.3,0.02)$ distributions, respectively. The dashed lines show the theoretic predictions from Conjecture C.
• Figure 3: Discretization via uniform $\Delta^{-1}=300$ partition of continuous random variable $X=S_n(Y)$ determined by variable $Y$ via the sine box function $S_n:x\mapsto \frac{1+\sin 2\pi n x}{2}$. In the left and middle panels, the scatter plots show the joint distribution $P_{X^{\Delta}Y^{\Delta}}$ of the discretized variables $X^{\Delta},Y^{\Delta}$, together with the marginal distributions $P_{Y^{\Delta}}$ at the top and $P_{X^{\Delta}}$ on the right. The right panel plots for each $n$, the corresponding MI $I(X^{\Delta};Y^{\Delta})$ of the discretized variables, with the empirical values shown in dots and Conjectured values in dashed lines. The blue and red colors correspond to $Y$ following the uniform and acip distributions, respectively.

Theorems & Definitions (28)

• Example 1.1
• Remark 1.2
• Remark 1.3
• Remark 1.4
• Example 1.5: Bernoulli interval maps
• Example 1.6: Sine box functions
• Remark 2.1
• Lemma 2.2: Divergence inequality, Gray2011 Lemma 5.2.1
• Lemma 2.3: Relative entropy density Gray2011 Lemma 5.2.3
• Remark 2.4