Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adjusted Similarity Measures and a Violation of Expectations

William L. Lippitt, Edward J. Bedrick, Nichole E. Carlson

TL;DR

The paper addresses how adjusted similarity measures for chance depend on the chosen null model and normalization. It proposes a generalized adjustment AS(n) = (S(n) - E_{M^n}[S(\tilde{n})]) / (S_{max}^n - E_{M^n}[S(\tilde{n})]), and derives sufficient constancy conditions under which the adjusted index has mean 0, with standardization yielding variance 1, and the adjustment operator being idempotent. It also demonstrates that data-driven null models can violate these properties, producing non-positive adjustments or identically zero standardizations, highlighting the special role of the permutation model. The results provide concrete guidelines for selecting null models and maxima to preserve intended interpretation, and identify clear directions for extending these ideas to broader clustering contexts and approximate asymptotics. Overall, the work offers a formal framework to reason about adjustment for chance beyond classical permutation-based approaches and cautions against uncritical use of generalized adjustments without constancy guarantees.

Abstract

Adjusted similarity measures, such as Cohen's kappa for inter-rater reliability and the adjusted Rand index used to compare clustering algorithms, are a vital tool for comparing discrete labellings. These measures are intended to have the property of 0 expectation under a null distribution and maximum value 1 under maximal similarity to aid in interpretation. Measures are frequently adjusted with respect to the permutation distribution for historic and analytic reasons. There is currently renewed interest in considering other null models more appropriate for context, such as clustering ensembles permitting a random number of identified clusters. The purpose of this work is two -- fold: (1) to generalize the study of the adjustment operator to general null models and to a more general procedure which includes statistical standardization as a special case and (2) to identify sufficient conditions for the adjustment operator to produce the intended properties, where sufficient conditions are related to whether and how observed data are incorporated into null distributions. We demonstrate how violations of the sufficient conditions may lead to substantial breakdown, such as by producing a non-positive measure under traditional adjustment rather than one with mean 0, or by producing a measure which is deterministically 0 under statistical standardization.

Adjusted Similarity Measures and a Violation of Expectations

TL;DR

The paper addresses how adjusted similarity measures for chance depend on the chosen null model and normalization. It proposes a generalized adjustment AS(n) = (S(n) - E_{M^n}[S(\tilde{n})]) / (S_{max}^n - E_{M^n}[S(\tilde{n})]), and derives sufficient constancy conditions under which the adjusted index has mean 0, with standardization yielding variance 1, and the adjustment operator being idempotent. It also demonstrates that data-driven null models can violate these properties, producing non-positive adjustments or identically zero standardizations, highlighting the special role of the permutation model. The results provide concrete guidelines for selecting null models and maxima to preserve intended interpretation, and identify clear directions for extending these ideas to broader clustering contexts and approximate asymptotics. Overall, the work offers a formal framework to reason about adjustment for chance beyond classical permutation-based approaches and cautions against uncritical use of generalized adjustments without constancy guarantees.

Abstract

Adjusted similarity measures, such as Cohen's kappa for inter-rater reliability and the adjusted Rand index used to compare clustering algorithms, are a vital tool for comparing discrete labellings. These measures are intended to have the property of 0 expectation under a null distribution and maximum value 1 under maximal similarity to aid in interpretation. Measures are frequently adjusted with respect to the permutation distribution for historic and analytic reasons. There is currently renewed interest in considering other null models more appropriate for context, such as clustering ensembles permitting a random number of identified clusters. The purpose of this work is two -- fold: (1) to generalize the study of the adjustment operator to general null models and to a more general procedure which includes statistical standardization as a special case and (2) to identify sufficient conditions for the adjustment operator to produce the intended properties, where sufficient conditions are related to whether and how observed data are incorporated into null distributions. We demonstrate how violations of the sufficient conditions may lead to substantial breakdown, such as by producing a non-positive measure under traditional adjustment rather than one with mean 0, or by producing a measure which is deterministically 0 under statistical standardization.
Paper Structure (12 sections, 4 theorems, 21 equations, 1 figure, 1 table)

This paper contains 12 sections, 4 theorems, 21 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. General Adjustment for Chance
  3. Adjustment of an Index
  4. Common Null Models and Normalizations
  5. Impacts of the Choice of Generalized Normalization Smax n
  6. General Adjustment as an Operator
  7. Favorable Properties Through Linear Families
  8. Violations of Intended Properties
  9. Examples of Violation
  10. Conclusion
  11. Future Work
  12. Proposition \ref{['prop:fail']} parts (3.), (4.), and (5.) allowing different conventions c for when the maximal index value is expected.

Key Result

Theorem 3.4

Given $T\in\mathcal{L}_{\mathcal{M}}(S)$ and $S_{max}$, define $T_{max}$ according to Equation eqn:pmax. Then the adjustment $AT(n)$ of $T(n)$ with respect to $\mathcal{M}$ and $T_{max}$ is equivalent to the adjustment $AS(n)$ of $S(n)$ with respect to $\mathcal{M}$ and $S_{max}$: $AT(n)=AS(n)$.

Figures (1)

  • Figure 1: Visualization of violation of idempotency, i.e., visualization of the general non-equivalence of $AS$ and $A^2S$, for an example index $S$. Specifically, for $S(n)=u_1^2$, $S_{max}=N^2$, $AS_{max}=\max(0,c)$, we have heatmaps of $-\log_{10}|AS(n)-A^2S(n)|$ as a function of $u_1$, $N$, and convention $c$ for the adjusted value of an index $S$ when $S_{max}=\text{E}_{M^n}\left[S(\tilde{n})\right]$. Larger log differences indicate approaching equivalence.

Theorems & Definitions (16)

  • Definition 2.1: Index $S$
  • Definition 2.2: Raw proportion $p$ of agreement
  • Definition 2.3: Proportion $q$ of pair agreement
  • Definition 2.4: Adjusted Index $AS$
  • Definition 2.5: $\mathcal{M}_{perm}$
  • Definition 2.6: $\mathcal{M}_{2,ind}$
  • Definition 2.7: $\mathcal{M}_{1,ind}$
  • Definition 3.1: Pair contingency table
  • Definition 3.2: $\mathcal{L}$; albatineh2006similarity
  • Definition 3.3: $\mathcal{L}_{\mathcal{M}}(S)$
  • ...and 6 more