Separable Computation of Information Measures

TL;DR

This work introduces a separable framework for estimating information measures by first learning feature representations $S=s(X)$ and $T=t(Y)$ that are sufficient statistics, then applying estimators on $(S,T)$. It proves that a broad class of information measures—including $I(X;Y)$, $I_f(X;Y)$, Wyner's common information, Gács–Körner common information, and the information bottleneck quantities—admits such separable computation, with invariance results $\theta(X,Y)=\theta(S,T)$. The analysis hinges on the canonical dependence kernel and its modal decomposition, and it shows that optimal constructions can be confined to the sufficiency-reduced space, providing theoretical guarantees for modular, feature-centric estimators in high-dimensional settings. The results connect information-measure estimation to the dependence structure and suggest principled feature-learning strategies (e.g., via universal features $f^*(X)$, $g^*(Y)$) for efficient, scalable estimation.

Abstract

We study a separable design for computing information measures, where the information measure is computed from learned feature representations instead of raw data. Under mild assumptions on the feature representations, we demonstrate that a class of information measures admit such separable computation, including mutual information, $f$-information, Wyner's common information, G{á}cs--K{ö}rner common information, and Tishby's information bottleneck. Our development establishes several new connections between information measures and the statistical dependence structure. The characterizations also provide theoretical guarantees of practical designs for estimating information measures through representation learning.

Figures (1)

• Figure 1: Computing an information measure $\theta(X, Y)$ in two steps: (1) obtain transformed variables $s(X), t(Y)$ and (2) apply an estimator on $s(X), t(Y)$. For high-dimensional $X$, $Y$ with unknown probability structures, the first step can be implemented by data-driven approaches, e.g., deep neural network training, corresponding to the feature learning process, where $s(X)$ and $t(Y)$ are the learned feature representations.

Theorems & Definitions (13)

• Definition 1
• Proposition 1
• Proposition 2: xu2024dependence
• Corollary 1
• Theorem 1
• Theorem 2
• Remark 1
• Theorem 3
• Lemma 1
• Lemma 2