Evaluation of Cluster Id Assignment Schemes with ABCDE
Stephan van Staden
TL;DR
The paper addresses the problem of evaluating cluster id assignment schemes with the goal of semantic id stability across clusterings. It recasts id assignment as a cluster-membership change problem and applies the ABCDE framework to quantify both the magnitude ($JaccardDistance$, $SplitRate$, $MergeRate$) and the semantic quality ($GoodSplitRate$, $BadSplitRate$, $ΔPrecision$, $ΔRecall$, $IQ$) of differences between a baseline and an experiment. It constructs inputs Base and Exp by combining historical membership with current clustering data and introduces item weights, along with several generalizations such as simultaneous membership and id changes, cross-item inputs, emphasis control via $HistScaleFactor$, and multiple historical epochs. The approach yields interpretable diagnostics and scales to real-world setups with billions of items and millions of clusters, guiding practitioners in selecting id-assignment schemes that preserve semantic identities.
Abstract
A cluster id assignment scheme labels each cluster of a clustering with a distinct id. The goal of id assignment is semantic id stability, which means that, whenever possible, a cluster for the same underlying concept as that of a historical cluster should ideally receive the same id as the historical cluster. Semantic id stability allows the users of a clustering to refer to a concept's cluster with an id that is stable across clusterings/time. This paper treats the problem of evaluating the relative merits of id assignment schemes. In particular, it considers a historical clustering with id assignments, and a new clustering with ids assigned by a baseline and an experiment. It produces metrics that characterize both the magnitude and the quality of the id assignment diffs between the baseline and the experiment. That happens by transforming the problem of cluster id assignment into a problem of cluster membership, and evaluating it with ABCDE. ABCDE is a sophisticated and scalable technique for evaluating differences in cluster membership in real-world applications, where billions of items are grouped into millions of clusters, and some items are more important than others. The paper also describes several generalizations to the basic evaluation setup for id assignment schemes. For example, it is fairly straightforward to evaluate changes that simultaneously mutate cluster memberships and cluster ids. The ideas are generously illustrated with examples.