Isotropy, Clusters, and Classifiers

TL;DR

The paper investigates the tension between isotropy in embedding spaces and the presence of clusters, showing that strict isotropy, as quantified by IsoScore, effectively forbids cluster structure and thus conflicts with linear classification. It formalizes this tension through a silhouette-based clustering objective and demonstrates that its optimal value aligns with a squared-distance surrogate, highlighting incompatibility with isotropy. The work combines mathematical derivations with empirical validation and reframes prior results on anisotropy by linking isotropy to embedding-space geometry and clustering. The findings have practical implications for evaluating and optimizing embeddings, clarifying when isotropy is desirable and how it may hinder linear separability in downstream tasks.

Abstract

Whether embedding spaces use all their dimensions equally, i.e., whether they are isotropic, has been a recent subject of discussion. Evidence has been accrued both for and against enforcing isotropy in embedding spaces. In the present paper, we stress that isotropy imposes requirements on the embedding space that are not compatible with the presence of clusters -- which also negatively impacts linear classification objectives. We demonstrate this fact both mathematically and empirically and use it to shed light on previous results from the literature.