Common Information Dimension

TL;DR

The paper introduces Common Information Dimension (CID) and its Rényi and Gács–Körner variants to quantify shared randomness for continuous variables under function-class constraints, addressing cases where entropy-based measures may be infinite. For jointly Gaussian vectors with linear function class, it derives closed-form CID/RCID and a constructive method to obtain the minimal-dimension common variable, linking the CID to covariance ranks. It then connects CID to Wyner's common information by showing that, in nearly singular, approximate, and quantized regimes, the growth of Wyner’s information is governed by the CID, with precise $ (1/2) ext{log}(1/ ext{ε})$ or $ ext{log} m$ scaling. The work provides both theoretical foundations and numerical validation, offering practical guidance for distributed simulation when dealing with continuous sources. These results illuminate how dimensionality constraints can effectively quantify the resources needed to simulate joint distributions in the continuous setting and bridge CID with classical information measures.

Abstract

The exact common information between a set of random variables $X_1,...,X_n$ is defined as the minimum entropy of a shared random variable that allows for the exact distributive simulation of $X_1,...,X_n$. It has been established that, in certain instances, infinite entropy is required to achieve distributive simulation, suggesting that continuous random variables may be needed in such scenarios. However, to date, there is no established metric to characterize such cases. In this paper, we propose the concept of Common Information Dimension (CID) with respect to a given class of functions $\mathcal{F}$, defined as the minimum dimension of a random variable $W$ required to distributively simulate a set of random variables $X_1,...,X_n$, such that $W$ can be expressed as a function of $X_1,\cdots,X_n$ using a member of $\mathcal{F}$. Our main contributions include the computation of the common information dimension for jointly Gaussian random vectors in a closed form, with $\mathcal{F}$ being the linear functions class.

Figures (3)

• Figure 1: The one-shot exact version of (a) Wyner's distributed simulation problem and (b) Gács-Körner's distributed randomness extraction problem.
• Figure 2: (a) the growth rate of $C_\epsilon(X,Y)$ and $C(X_\epsilon,Y_\epsilon)$; (b) the growth rate of $C(\langle X\rangle_m, \langle Y \rangle_m)$.
• Figure 3: common information dimension $d(X,Y)$ vs (a) the $\epsilon$-approximate common information $C_\epsilon(X,Y)$; (b) the common information of quantized variables $C(\langle X\rangle_m,\langle Y\rangle_m)$.

Theorems & Definitions (38)

• Definition 1
• Definition 2
• Definition 3
• Theorem 1
• Theorem 2
• Lemma 1
• Theorem 3
• Lemma 2
• Lemma 3
• Lemma 4