Distance-to-Distance Ratio: A Similarity Measure for Sentences Based on Rate of Change in LLM Embeddings
Abdullah Qureshi, Kenneth Rice, Alexander Wolpert
TL;DR
The paper addresses the challenge of measuring semantic similarity in LLM embeddings beyond traditional cosine-based approaches, which can miss subtle contextual shifts. It introduces Distance-to-Distance Ratio (DDR), a Lipschitz-inspired metric that compares the maximum pre-context token distances to the maximum post-context distances after LLM transformation, formalized as $\mathrm{DDR}(t,t') = \frac{d_{\text{in}}(x,x')}{d_{\text{out}}(F(x),F(x'))}$ with $d_{\text{in}}$ and $d_{\text{out}}$ defined via $d_{\text{chord}}$, and evaluates it against Centroid and EOS baselines. The study performs perturbation-based experiments on 500 text excerpts with one-to-three word edits, contrasting synonym and random substitutions; results show DDR provides finer discrimination across edit depths, outperforming Centroid and EOS in distinguishing semantically similar vs dissimilar edits. This work suggests DDR as a robust tool for evaluating semantic shifts, with practical implications for information retrieval, model robustness, and embedding method comparisons in high-dimensional contexts, particularly where small lexical edits should have predictable, contractive effects on representations. DDR’s reliance on ratio-based estimates also aligns with modern statistical insights that such ratios can be more stable to compute than absolute quantities in high-dimensional spaces.
Abstract
A measure of similarity between text embeddings can be considered adequate only if it adheres to the human perception of similarity between texts. In this paper, we introduce the distance-to-distance ratio (DDR), a novel measure of similarity between LLM sentence embeddings. Inspired by Lipschitz continuity, DDR measures the rate of change in similarity between the pre-context word embeddings and the similarity between post-context LLM embeddings, thus measuring the semantic influence of context. We evaluate the performance of DDR in experiments designed as a series of perturbations applied to sentences drawn from a sentence dataset. For each sentence, we generate variants by replacing one, two, or three words with either synonyms, which constitute semantically similar text, or randomly chosen words, which constitute semantically dissimilar text. We compare the performance of DDR with other prevailing similarity metrics and demonstrate that DDR consistently provides finer discrimination between semantically similar and dissimilar texts, even under minimal, controlled edits.