Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets

Kyungeun Lee, Wonjong Rhee

TL;DR

This study introduces a comprehensive benchmark suite for evaluating neural MI estimators on unstructured datasets, specifically focusing on images and texts, and shows that it can accurately manipulate true MI values of real-world datasets.

Abstract

Mutual Information (MI) is a fundamental metric for quantifying dependency between two random variables. When we can access only the samples, but not the underlying distribution functions, we can evaluate MI using sample-based estimators. Assessment of such MI estimators, however, has almost always relied on analytical datasets including Gaussian multivariates. Such datasets allow analytical calculations of the true MI values, but they are limited in that they do not reflect the complexities of real-world datasets. This study introduces a comprehensive benchmark suite for evaluating neural MI estimators on unstructured datasets, specifically focusing on images and texts. By leveraging same-class sampling for positive pairing and introducing a binary symmetric channel trick, we show that we can accurately manipulate true MI values of real-world datasets. Using the benchmark suite, we investigate seven challenging scenarios, shedding light on the reliability of neural MI estimators for unstructured datasets.

A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets

TL;DR

This study introduces a comprehensive benchmark suite for evaluating neural MI estimators on unstructured datasets, specifically focusing on images and texts, and shows that it can accurately manipulate true MI values of real-world datasets.

Abstract

Mutual Information (MI) is a fundamental metric for quantifying dependency between two random variables. When we can access only the samples, but not the underlying distribution functions, we can evaluate MI using sample-based estimators. Assessment of such MI estimators, however, has almost always relied on analytical datasets including Gaussian multivariates. Such datasets allow analytical calculations of the true MI values, but they are limited in that they do not reflect the complexities of real-world datasets. This study introduces a comprehensive benchmark suite for evaluating neural MI estimators on unstructured datasets, specifically focusing on images and texts. By leveraging same-class sampling for positive pairing and introducing a binary symmetric channel trick, we show that we can accurately manipulate true MI values of real-world datasets. Using the benchmark suite, we investigate seven challenging scenarios, shedding light on the reliability of neural MI estimators for unstructured datasets.

Paper Structure

This paper contains 29 sections, 6 theorems, 15 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Backgrounds: Neural Mutual Information Estimators
  3. Related Works
  4. Proposed Method and Benchmark Suite
  5. General formulation for Gaussian dataset
  6. Definitions of ds, dr, and Z
  7. Theoretical background: same-class sampling for positive pairing
  8. Generating datasets with adjustable true MI values
  9. Manipulating MI to non-integer values: binary symmetric channel
  10. Empirical investigations
  11. Choice of critic architecture: superiority of joint critic for unstructured datasets
  12. Choice of critic capacity: larger capacity does not ensure a higher estimation accuracy
  13. Choice of MI estimator: no universal winner exists across the three data domains
  14. Number of information sources (ds): unstructured datasets outperform Gaussian in handling larger ds
  15. Representation dimension (dr): it does not affect the estimation accuracy
  16. ...and 14 more sections

Key Result

Theorem 4.4

When the information source $C$ is transmitted perfectly to $X$, while it is transmitted to $Y$ over a binary symmetric channel (BSC) with a crossover probability $\beta \in [0, 0.5]$, the mutual information $I(X;Y)$ between $X$ and $Y$ is determined as follows.

Figures (12)

  • Figure 1: Exemplary findings from our benchmark suite. The ratio between estimated MI value and true MI value is shown in y-axis. A ratio close to 1.0 indicates a highly accurate MI estimation. (a, b, c) When the number of independent information sources is increased to 16, all MI estimators become inaccurate for Gaussian dataset (shown in (a)) but some MI estimators remain accurate for image dataset (shown in (b)) and text dataset (shown in (c)). (d) When MI is estimated using embeddings from different layers of ResNet-50, MI estimators stay accurate only for the upper layers.
  • Figure 2: Four image examples of generating datasets with known true MI values. $X$ and $Y$ consist of random images drawn from the MNIST dataset. (a) Basic construction: only the images of class 0 or 1 are considered and $X$ and $Y$ are sampled to share the class information. By choosing $C$ to be either 0 or 1 with probability $0.5$, $H(C)$ becomes 1 and therefore $I(X;Y)=H(C)=1$. (b) Combining four independent images in 2-D to form a single image: $I(X;Y)=4$ because $H(C)=H_\text{left,up}+H_\text{left,down}+H_\text{right,up}+H_\text{right,down}=4$. (c) Combining three independent images in channel dimension to form a color image: $I(X;Y)=3$ because $H(C)=H_\text{green}+H_\text{red}+H_\text{blue}=3$. (d) Adding nuisance: an independently chosen background image from CIFAR-10 cifar is inserted as nuisance. Because the nuisance is independently chosen for $X$ and $Y$, they do not affect the true MI lee2023towards. Therefore, $I(X;Y)=1$.
  • Figure 3: Estimation results for four different benchmarks with $d_s=10$. Following the experimental setup of poole2019variational, we change the true MI values stepwise and the other hyperparameters are fixed.
  • Figure 4: Estimation results varying $d_s$. Shades correspond to the standard deviation of the estimations.
  • Figure 5: (a) Example of inserting nuisance to $\mathcal{D}_\text{vision}$. (b) Estimation results when the true MI is 2 bits. (c) Estimation results with various values of nuisance strength and true MI for three best-performing estimators. True MI values are on the x-axis and the nuisance strength is on the y-axis.
  • ...and 7 more figures

Theorems & Definitions (15)

  • Definition 2.1: Variational MI estimators poole2019variational
  • Definition 4.1: Number of information sources $d_s$
  • Definition 4.2: Representation dimension $d_r$
  • Definition 4.3: Nuisance $Z$
  • Theorem 4.4: Manipulating MI to be non-integer
  • Proposition B.1: InfoNCE estimation as a lower bound of the true MI oord2018cpcpoole2019variational
  • Proposition B.2: $\log{(2K-1)}$ Bound oord2018cpcpoole2019variational
  • proof
  • Proposition B.3
  • proof
  • ...and 5 more