Calibrated Similarity for Reliable Geometric Analysis of Embedding Spaces

TL;DR

This work shows that cosine similarity in pretrained embedding spaces is strong for ranking but suffers from anisotropy that miscalibrates absolute similarity values. It proposes isotonic regression as a monotone calibration of the similarity score, preserving ranking while aligning scores with human judgments (achieving $ECE\approx 0$ and $MBE=0$) and maintaining high local stability ($98\%$ across seven perturbations). Theoretical results establish that isotonic calibration preserves angular order, nearest-neighbor relations, threshold graphs, and high-confidence thresholds, enabling calibrated similarity to serve as a drop-in replacement for geometric analyses. Empirically, calibrated similarity (with $\tilde{s}$) improves interpretability for thresholding, clustering, retrieval, and cross-model evaluation, without modifying the underlying embeddings, and a principled high-confidence threshold $\tilde{\tau}_{\text{HCS}}$ supports robust decision making in practical tasks.

Abstract

While raw cosine similarity in pretrained embedding spaces exhibits strong rank correlation with human judgments, anisotropy induces systematic miscalibration of absolute values: scores concentrate in a narrow high-similarity band regardless of actual semantic relatedness, limiting interpretability as a quantitative measure. Prior work addresses this by modifying the embedding space (whitening, contrastive fine tuning), but such transformations alter geometric structure and require recomputing all embeddings. Using isotonic regression trained on human similarity judgments, we construct a monotonic transformation that achieves near-perfect calibration while preserving rank correlation and local stability(98% across seven perturbation types). Our contribution is not to replace cosine similarity, but to restore interpretability of its absolute values through monotone calibration, without altering its ranking properties. We characterize isotonic calibration as an order-preserving reparameterization and prove that all order-based constructions (angular ordering, nearest neighbors, threshold graphs and quantile-based decisions) are invariant under this transformation.

Figures (16)

• Figure 1: Density plot (KDE) of original (uncalibrated) cosine similarity versus human similarity scores.
• Figure 2: Heatmap of original (uncalibrated) cosine similarity versus human similarity scores.
• Figure 3: Linear regression calibration: density plot (KDE).
• Figure 4: Linear regression calibration: heatmap.
• Figure 5: Isotonic regression calibration: density plot (KDE).