Vendi Information Gain: An Alternative To Mutual Information For Science And Machine Learning

TL;DR

Shannon MI has known limitations in high dimensions, estimation from samples, and symmetry. The authors propose Vendi Information Gain (VIG), defined as $\mathrm{VIG}(\theta,y; q) = H_V(D; q) - \mathbb{E}_y[H_V(D_y; q)]$, where $H_V$ is the Vendi entropy computed from kernel eigenvalues. VIG generalizes MI and is asymmetric, requiring only samples and a similarity kernel, reducing to MI when samples are completely dissimilar. Across cognitive-science and epidemiology tasks, VIG provides a unified information-gain framework for active data acquisition and level-set estimation and consistently outperforms MI. Code and data are released to enable replication.

Abstract

In his 1948 seminal paper A Mathematical Theory of Communication that birthed information theory, Claude Shannon introduced mutual information (MI), which he called "rate of transmission", as a way to quantify information gain (IG) and defined it as the difference between the marginal and conditional entropy of a random variable. While MI has become a standard tool in science and engineering, it has several shortcomings. First, MI is often intractable - it requires a density over samples with tractable Shannon entropy - and existing techniques for approximating it often fail, especially in high dimensions. Moreover, in settings where MI is tractable, its symmetry and insensitivity to sample similarity are undesirable. In this paper, we propose the Vendi Information Gain (VIG), a novel alternative to MI that leverages the Vendi scores, a flexible family of similarity-based diversity metrics. We call the logarithm of the VS the Vendi entropy and define VIG as the difference between the marginal and conditional Vendi entropy of a variable. Being based on the VS, VIG accounts for similarity. Furthermore, VIG generalizes MI and recovers it under the assumption that the samples are completely dissimilar. Importantly, VIG only requires samples and not a probability distribution over them. Finally, it is asymmetric, a desideratum for a good measure of IG that MI fails to meet. VIG extends information theory to settings where MI completely fails. For example, we use VIG to describe a novel, unified framework for active data acquisition, a popular paradigm of modern data-driven science. We demonstrate the advantages of VIG over MI in diverse applications, including in cognitive science to model human response times to external stimuli and in epidemiology to learn epidemic processes and identify disease hotspots in different countries via level-set estimation.

Figures (10)

• Figure 1: Illustration of MI's failure modes and VIG's benefits. MI estimates tend to become biased in unpredictable ways as the dimensionality increases in the second column and degenerate to $0$ with fewer samples in the third column, while VIG estimates are relatively stable in both. Last column: Compared to MI, VIG better corresponds to the trend of the average predictive error conditioned on the label.
• Figure 2: Simulated response times as a function of similarity between possible messages by MI and VIG against observed experimental data from slamecka1963choice. MI's insensitivity to inter-message similarity prevents the simulation of different behaviors under different conditions, while VIG's simulations closely match observed real-world data.
• Figure 3: Illustration of Vendi information gain for active data acquisition. For each candidate $x$ we may query, we first draw Monte Carlo samples of its label $\{ y_i \}$ according to our predictive model trained on observed data $f_\mathcal{D}$. Conditioned on each sample, we then draw fantasized samples of all labels within the search space $\boldsymbol{\xi}_i$, which are transformed by function $g(\cdot)$ to yield samples of the quantity of interest $\boldsymbol{\theta}_i$. Information gain is then computed on each set of samples, and the average across different fantasized labels gives the expected VIG.
• Figure 4: Average estimation error ($\pm$ 1 standard error) across 50 repeats. VIG is competitive against MI, at times outperforming MI by a large margin.
• Figure 5: Average F$_1$ scores and standard errors achieved by LSE policies as a function of the number of queries. MI performs the worst, while VIG consistently achieves the highest F$_1$ scores, sometimes significantly outperforming baselines.