Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unifying Information-Theoretic and Pair-Counting Clustering Similarity

Alexander J. Gates

TL;DR

This work develops an analytical framework that unifies pair-counting and information-theoretic clustering similarity measures by expressing both as dependence-function expansions around an independence baseline. It reveals a quadratic core weighted by residuals $\\delta_{ij}$, with higher-order terms capturing sharper or localized structure, and shows that information-based scores emphasize minority, high-information coincidences while pair-based scores emphasize large-cluster agreement. A Rényi-entropy, collision-probability viewpoint introduces $I^{(k)}$ estimators that interpolate between pair-counting ($k=2$) and full MI (large $k$), providing a tunable continuum of similarity measures and clarifying when indices diverge. The framework also connects these ideas to element-centric similarity via random-walk paths, yielding practical guidance for selecting metrics, normalizing, and extending to more complex clustering scenarios, with illustrative examples highlighting the balance between robustness and sensitivity across regimes.

Abstract

Comparing clusterings is central to evaluating unsupervised models, yet the many existing similarity measures can produce widely divergent, sometimes contradictory, evaluations. Clustering similarity measures are typically organized into two principal families, pair-counting and information-theoretic, reflecting whether they quantify agreement through element pairs or aggregate information across full cluster contingency tables. Prior work has uncovered parallels between these families and applied empirical normalization or chance-correction schemes, but their deeper analytical connection remains only partially understood. Here, we develop an analytical framework that unifies these families through two complementary perspectives. First, both families are expressed as weighted expansions of observed versus expected co-occurrences, with pair-counting arising as a quadratic, low-order approximation and information-theoretic measures as higher-order, frequency-weighted extensions. Second, we generalize pair-counting to $k$-tuple agreement and show that information-theoretic measures can be viewed as systematically accumulating higher-order co-assignment structure beyond the pairwise level. We illustrate the approaches analytically for the Rand index and Mutual Information, and show how other indices in each family emerge as natural extensions. Together, these views clarify when and why the two regimes diverge, relating their sensitivities directly to weighting and approximation order, and provide a principled basis for selecting, interpreting, and extending clustering similarity measures across applications.

Unifying Information-Theoretic and Pair-Counting Clustering Similarity

TL;DR

This work develops an analytical framework that unifies pair-counting and information-theoretic clustering similarity measures by expressing both as dependence-function expansions around an independence baseline. It reveals a quadratic core weighted by residuals , with higher-order terms capturing sharper or localized structure, and shows that information-based scores emphasize minority, high-information coincidences while pair-based scores emphasize large-cluster agreement. A Rényi-entropy, collision-probability viewpoint introduces estimators that interpolate between pair-counting () and full MI (large ), providing a tunable continuum of similarity measures and clarifying when indices diverge. The framework also connects these ideas to element-centric similarity via random-walk paths, yielding practical guidance for selecting metrics, normalizing, and extending to more complex clustering scenarios, with illustrative examples highlighting the balance between robustness and sensitivity across regimes.

Abstract

Comparing clusterings is central to evaluating unsupervised models, yet the many existing similarity measures can produce widely divergent, sometimes contradictory, evaluations. Clustering similarity measures are typically organized into two principal families, pair-counting and information-theoretic, reflecting whether they quantify agreement through element pairs or aggregate information across full cluster contingency tables. Prior work has uncovered parallels between these families and applied empirical normalization or chance-correction schemes, but their deeper analytical connection remains only partially understood. Here, we develop an analytical framework that unifies these families through two complementary perspectives. First, both families are expressed as weighted expansions of observed versus expected co-occurrences, with pair-counting arising as a quadratic, low-order approximation and information-theoretic measures as higher-order, frequency-weighted extensions. Second, we generalize pair-counting to -tuple agreement and show that information-theoretic measures can be viewed as systematically accumulating higher-order co-assignment structure beyond the pairwise level. We illustrate the approaches analytically for the Rand index and Mutual Information, and show how other indices in each family emerge as natural extensions. Together, these views clarify when and why the two regimes diverge, relating their sensitivities directly to weighting and approximation order, and provide a principled basis for selecting, interpreting, and extending clustering similarity measures across applications.

Paper Structure

This paper contains 17 sections, 34 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Notation
  3. Clusterings
  4. Clustering similarity and single elements
  5. From single elements to unordered pairs
  6. Independent Clusterings
  7. Bridging Mutual Information and Pair-Counting via Expansion Around Independence
  8. Expanding mutual information
  9. Connection to pair-counting indices
  10. Why not keep adding residual terms?
  11. Rényi entropies as pair–counting in disguise
  12. Collision Probability and Rényi Entropies
  13. Approximating Mutual Information with Higher–Order Rényi Contrasts
  14. From Collisions to Structure: Insights from $k$–tuple Approximations
  15. Linking collision probabilities and element–centric similarity
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Balanced clustering similarity.$N=1,000$ elements are grouped into a balanced clustering $\mathcal{A}$ with two clusters of size $500$, while $\mathcal{B}$ is made by exchanging the membership of a fraction of the elements, $\epsilon\in[0,0.5]$, from $\mathcal{A}$. A) For each $\epsilon$ we compute the Rand Index (RI), Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), and normalized variants of the $I^{(2)}$, $I^{(3)}$, and $I^{(4)}$ Rényi contrasts approximations to MI. Curves represent the average over 100 independent trials, while shaded area reflects two standard errors of the mean. B) Residual matrices (normalized to highlight relative magnitudes) for the MI and ARI between $\mathcal{A}$ and $\mathcal{B}$ with exchange level $\epsilon=0.25$.
  • Figure 2: Balanced clustering similarity.$N=1,000$ elements are grouped into an unbalanced clustering $\mathcal{A}$ with one big cluster of size $800$ and two small clusters of $100$ elements each, while clustering $\mathcal{B}$ is made by exchanging the membership of a fraction of the elements, $\epsilon\in[0,0.5]$, from $\mathcal{A}$ between the (A) "small-small" clusters or (B) "big-small" clusters. For each $\epsilon$ we compute the Rand Index (RI), Adjusted Rand Index (ARI), Normalized Mutual Information (NMI), and normalized variants of the $I^{(2)}$, $I^{(3)}$, and $I^{(4)}$ Rényi contrasts approximations to MI. Curves represent the average over 100 independent trials, while shaded area reflects two standard errors of the mean. C-D) Residual matrices (normalized to highlight relative magnitudes)for the MI and ARI between $\mathcal{A}$ and $\mathcal{B}$ with exchange level $\epsilon=0.5$ for the C small-small and D big-small exchange examples.