Pointwise Metrics for Clustering Evaluation

TL;DR

This paper introduces pointwise clustering metrics, a weighted, per-item framework for evaluating how an actual clustering aligns with a ground-truth clustering. It defines per-item confusion components and metrics (e.g., $Precision(i)$, $Recall(i)$, $JaccardDistance(i)$), aggregates them via weighted averages, and proves that the lifted per-item distance $d'(C_1,C_2)$ is a true metric on clusterings. The approach supports analysis at multiple granularities (items, clusters, slices) and facilitates delta-based comparisons between algorithms. It also situates the method relative to co-membership and cross-tabulation metrics, showing that the pointwise formulation can reproduce or extend existing measures like ARAND, F-measure, B-CUBED, and V-measure. The framework is practical for large-scale, evolving datasets and provides a solid mathematical foundation for scalable clustering evaluation with ground-truth references.

Abstract

This paper defines pointwise clustering metrics, a collection of metrics for characterizing the similarity of two clusterings. These metrics have several interesting properties which make them attractive for practical applications. They can take into account the relative importance of the various items that are clustered. The metric definitions are based on standard set-theoretic notions and are simple to understand. They characterize aspects that are important for typical applications, such as cluster homogeneity and completeness. It is possible to assign metrics to individual items, clusters, arbitrary slices of items, and the overall clustering. The metrics can provide deep insights, for example they can facilitate drilling deeper into clustering mistakes to understand where they happened, or help to explore slices of items to understand how they were affected. Since the pointwise metrics are mathematically well-behaved, they can provide a strong foundation for a variety of clustering evaluation techniques. In this paper we discuss in depth how the pointwise metrics can be used to evaluate an actual clustering with respect to a ground truth clustering.

Figures (3)

• Figure 1: The Venn diagram of the 2x2 clustering confusion matrix from the perspective of item $i$. The item $i$ is always in the intersection of $\mathit{IdealCluster}(i)$ and $\mathit{ActualCluster}(i)$. Hence $\mathit{weight}(i) \subseteq \mathit{TP}(i)$, so the the weight of $i$ is contained in the intersection, which is always non-empty. The left circle is labeled with Ideal, which is a shorthand for $\mathit{weight}(\mathit{IdealCluster}(i))$. The right circle is labeled with Actual, which is a shorthand for $\mathit{weight}(\mathit{ActualCluster}(i))$.
• Figure 2: Pointwise clustering metrics in action.
• Figure 3: Clusterings like those of Figure \ref{['small_clustering_example']} but where the items have been subdivided: $i_2$ was divided into $i_4$ and $i_5$, while $i_3$ was divided into $i_6$, $i_7$ and $i_8$. If the new items all have weight 1, then overall $\mathit{Precision} = 3/4$, overall $\mathit{Recall} = 7/9$ and overall $\mathit{JaccardDistance} = 3/8$, as was the case in Figure \ref{['small_clustering_example']}.

Theorems & Definitions (8)

• Lemma 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Theorem 4
• proof