Neural Mutual Information Estimation with Vector Copulas

TL;DR

This work tackles the fundamental problem of mutual information (MI) estimation with limited data by introducing a vector copula-based divide-and-conquer estimator (VCE). It partitions the task into learning the marginals with flow-based models and learning a flexible vector copula over the vector ranks, establishing that $I(X;Y) = -H[c(oldsymbol{u}_X,oldsymbol{u}_Y)]$, i.e., MI is the vector-copula entropy. The authors provide consistency and error analyses, reinterpret existing MI estimators through the vector-copula lens, and demonstrate robust performance across synthetic benchmarks, image-data MI-with-known-values, and language-model embeddings, often outperforming state-of-the-art discriminative and some generative methods. The approach offers a principled capacity–complexity trade-off via a mixture of vector copulas, enables reuse of learned marginals, and benefits from explicit marginal-copula separation, with practical implications for representation learning, data analysis, and fairness/privacy research where reliable MI estimates are crucial.

Abstract

Estimating mutual information (MI) is a fundamental task in data science and machine learning. Existing estimators mainly rely on either highly flexible models (e.g., neural networks), which require large amounts of data, or overly simplified models (e.g., Gaussian copula), which fail to capture complex distributions. Drawing upon recent vector copula theory, we propose a principled interpolation between these two extremes to achieve a better trade-off between complexity and capacity. Experiments on state-of-the-art synthetic benchmarks and real-world data with diverse modalities demonstrate the advantages of the proposed estimator.

Figures (11)

• Figure 1: Overview of the proposed vector copula-based MI estimator (VCE), which explicitly disentangles the modeling of marginal distribution and dependence structure (i.e., the vector copula). VCE first respectively computes the vector ranks $\mathbf{u}_X$ and $\mathbf{u}_Y$ corresponding to the two marginal variables $X$ and $Y$ with flow models ( ①). It then finds the vector copula $c \in \mathcal{C}$ from the vector copula pool $\mathcal{C}$ that best matches with the joint distribution $p(\mathbf{u}_X, \mathbf{u}_Y)$ of the estimated vector ranks ( ②). Mutual information $I(X; Y)$ is computed as the negative differential entropy of the vector copula $c$, which itself is irrelevant to the two marginal distributions $p(\mathbf{x})$ and $p(\mathbf{y})$.
• Figure 2: Comparing MI estimators under various dependence strengths $\rho$. Data in cases (b)(c) are generated by first sampling $X, Y \sim \mathcal{N}$ as in case (a), then transforming them with the shown transformations. The dimensionalities of the data in the five cases are 64, 32, 32, 32, 2 respectively.
• Figure 3: Comparing different MI estimators under various data dimensionality $d$ and fixed dependence levels. MoG corresponds to mixture of Gaussians. Spiral corresponds to spiral transformation.
• Figure 4: The image dataset butakov2024information, which contains images of rectangles and Gaussian plates.
• Figure 5: Additional results for the VCE' estimator. In this work, we implement VCE' by taking the reference copula as the independent copula $c'$, so that VCE' is equivalent to performing MINE in the vector copula space. Here MoT stands for 'mixture of triangles'.

Theorems & Definitions (19)

• Definition 1: Vector rank
• Definition 2: Vector copula
• Theorem 1: Vector Sklar Theorem
• Theorem 2: MI is vector copula entropy
• Proposition 1: Consistency of VCE
• Proposition 2: Error of vector copula-based MI estimate
• Proposition 3: Vector Gaussian copula as second-order approximation
• Proposition 4: Vector copula of independent RVs
• Theorem 2: MI is vector copula entropy
• Lemma 1: Equivalence between $p(\mathbf{u}_X, \mathbf{u}_Y)$ and $c(\mathbf{u}_X, \mathbf{u}_Y)$