CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement

TL;DR

CASE addresses the challenge of context-sensitive semantic textual similarity by pairing LLM-based conditioning with a lightweight, supervised projection. It encodes the condition given a sentence with $f(c; I(s))$, subtracts the unconditional condition embedding $f(c; I( emptyset))$, and learns a small FFN to produce low-dimensional CASE embeddings, optimizing cosine similarity to human C-STS ratings. Empirically, CASE improves over prior C-STS methods, with LLM embeddings outperforming MLMs, and achieves similar performance to fine-tuned encoders while reducing dimensionality to $512$ and requiring far less training compute. The approach yields higher isotropy in the embedding space and demonstrates practical gains for large-scale retrieval and QA tasks where context-aware similarity matters.

Abstract

The meaning conveyed by a sentence often depends on the context in which it appears. Despite the progress of sentence embedding methods, it remains unclear how to best modify a sentence embedding conditioned on its context. To address this problem, we propose Condition-Aware Sentence Embeddings (CASE), an efficient and accurate method to create an embedding for a sentence under a given condition. First, CASE creates an embedding for the condition using a Large Language Model (LLM), where the sentence influences the attention scores computed for the tokens in the condition during pooling. Next, a supervised nonlinear projection is learned to reduce the dimensionality of the LLM-based text embeddings. We show that CASE significantly outperforms previously proposed Conditional Semantic Textual Similarity (C-STS) methods on an existing standard benchmark dataset. We find that subtracting the condition embedding consistently improves the C-STS performance of LLM-based text embeddings. Moreover, we propose a supervised dimensionality reduction method that not only reduces the dimensionality of LLM-based embeddings but also significantly improves their performance.

Figures (6)

• Figure 1: The two conditions focus on different information described in the two sentences. Human annotators rate the two sentences 1--5, indicating a high-level (5) of semantic textual similarity under $c_{\rm high}$ than $c_{\rm low}$ (1). Our proposed condition-aware sentence embedding (CASE) method reports similarity scores that are well-aligned with the human similarity ratings.
• Figure 2: Overview of CASE. An LLM is prompted with $I(s)$ to obtain two separate embeddings $f(c;I(s_1))$ and $f(c;I(s_2))$ for the same condition $c$ for the two sentences $s_1$ and $s_2$. The unconditional embedding $f(c;I(\emptyset))$ is then computed using the prompt $I(\emptyset)$ and subtracted from each of those conditional embeddings. Finally, the embeddings are projected to a lower-dimensional space using a supervised FFN and their cosine similarity is computed as the C-STS rating.
• Figure 3: Spearman correlation on test set for different models over training steps with dimensionality 512. The $y$-axes of both subfigures are aligned, facilitating a direct comparison of the Spearman coefficients across the two line charts, with the same colour for the same model. Best viewed in colour.
• Figure 4: Spearman correlation coefficients of our CASE for NV embeddings on the test set over different dimensionalities. The horizontal dashed line represents the performance of fine-tuned SimCSE-base (768-dimensional).
• Figure 5: Embeddings-to-mean cosine similarity distributions across three LLM-based models. Each row compares cond - c and cond representations.