Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem

TL;DR

This paper generalizes Hu's information diagram theorem from Shannon entropy to a broad class of information measures by modeling random variables as elements of a commutative, idempotent monoid that acts on a group of conditionable functions. The core result is a generalized Hu theorem: from a base function $F_1$ satisfying a chain rule $F_1(XY)=F_1(X)+X.F_1(Y)$, higher-degree terms $F_q$ can be constructed and represented via a $G$-valued measure on atoms of a generalized Venn diagram, yielding structured decompositions that extend to Tsallis entropy, KL divergence, cross-entropy, submodular information functions, and machine-learning generalization error. The framework is then instantiated for Kolmogorov complexity (including Chaitin's prefix-free and plain variants), proving a version of Hu's theorem and linking expected interaction complexity to interaction information, with asymptotic per-bit equality in well-behaved limits. Overall, the work provides a unifying, abstract mechanism to derive information-diagram decompositions across a wide spectrum of information measures, fostering connections between algorithmic and classical information theory and suggesting numerous directions for future generalizations and applications.

Abstract

In information theory, one major goal is to find useful functions that summarize the amount of information contained in the interaction of several random variables. Specifically, one can ask how the classical Shannon entropy, mutual information, and higher interaction information relate to each other. This is answered by Hu's theorem, which is widely known in the form of information diagrams: it relates shapes in a Venn diagram to information functions, thus establishing a bridge from set theory to information theory. In this work, we view random variables together with the joint operation as a monoid that acts by conditioning on information functions, and entropy as a function satisfying the chain rule of information. This abstract viewpoint allows to prove a generalization of Hu's theorem. It applies to Shannon and Tsallis entropy, (Tsallis) Kullback-Leibler Divergence, cross-entropy, Kolmogorov complexity, submodular information functions, and the generalization error in machine learning. Our result implies for Chaitin's Kolmogorov complexity that the interaction complexities of all degrees are in expectation close to Shannon interaction information. For well-behaved probability distributions on increasing sequence lengths, this shows that the per-bit expected interaction complexity and information asymptotically coincide, thus showing a strong bridge between algorithmic and classical information theory.

Figures (13)

• Figure 1: The generalized Hu theorem, visualized for a commutative, idempotent monoid $M$ generated by $X, Y$, and for $F_1$ and $F_2$. The measure $\mu$ turns sets into elements of the abelian group $G$ and disjoint unions into sums.
• Figure 2: A visualization of the generalized Hu theorem for a commutative, idempotent monoid generated by $X_1, X_2, X_3$. On the left-hand-side, three subsets of the abstract set $\widetilde{X}$ are emphasized, namely $\widetilde{X}_{12} \cap \widetilde{X}_{13}$, $\widetilde{X}_1 \setminus \widetilde{X}_3$, and $\widetilde{X}_{12} \cap \widetilde{X}_3$. On the right-hand-side, Equation \ref{['eq:hu_kuo_ting_equation']} turns them into elements of the abelian group $G$, namely $F_2(X_{12}; X_{13})$, $X_3.F_1(X_1)$, and $F_2(X_{12}; X_3)$, respectively. Many decompositions of information functions into sums directly follow from the theorem by using that $\mu$ turns disjoint unions into sums, as exemplified by the equation $F_2(X_{12};X_{13}) = X_3.F_1(X_1) + F_2(X_{12}; X_3)$.
• Figure 3: A visualization of the generalized Hu theorem for a commutative, idempotent monoid $M$ generated by $X_1, X_2, X_3, X_4$. To reduce clutter, we restrict to a visualization of the abstract sets $\widetilde{X}_i$ and the atoms $p_I$, as well as the corresponding information functions. On the right-hand-side, for computing $\mu(p_I)$ for the $15$ atoms $p_I$, we use Lemma \ref{['lem:p_I_charac']}.
• Figure 4: A visualization of Hu's theorem for Kolmogorov complexity for three variables $X, Y, Z$. On the left-hand-side, three subsets of the abstract set $\widetilde{XYZ}$ are emphasized, namely $\widetilde{XY} \cap \widetilde{XZ}$, $\widetilde{X} \setminus \widetilde{Z}$, and $\widetilde{XY} \cap \widetilde{Z}$. On the right-hand-side, Equation \ref{['eq:equality_hkt_chaitin_kolm']} turns them up to a constant error into the Kolmogorov complexity terms $Kc_2(XY; XZ)$, $Kc(X \mid Z)$, and $Kc_2(XY; Z)$, respectively. Many decompositions of complexity terms into sums directly follow from the theorem by using that $\mu$ turns disjoint unions into sums, as exemplified by the equation $Kc_2(XY;XZ) \overset{+}{=} Kc(X \mid Z) + Kc_2(XY; Z)$.
• Figure 5: Binary symmetric channels for the joint distributions $P$ and $Q$ in Example \ref{['exa:binary_symmetric_channel']}. For a uniform prior $P(X) = Q(X)$, $P$ and $Q$ have the same marginals $P(Y) = Q(Y)$, but differ in their conditionals $P(Y \mid X)$ and $Q(Y \mid X)$. This leads for small $\epsilon > 0$ to an arbitrarily large negative mutual Kullback-Leibler divergence $[D_2(X;Y)](P \| Q)$.

Theorems & Definitions (79)

• Definition 2.1: Shannon Entropy
• Definition 2.2: Entropy Function of a Random Variable
• Definition 2.3: Conditionable Functions, Averaged Conditioning
• Lemma 2.4
• Corollary 2.5
• Proposition 2.6
• proof
• Definition 2.7: Mutual Information, Interaction Information
• Remark 2.8
• Proposition 2.9