Sentence Smith: Controllable Edits for Evaluating Text Embeddings

TL;DR

SentenceSmith presents a neuro-symbolic framework that maps sentences to a semantic graph (AMR), applies human-designed, controllable manipulations, and regenerates text with a faithfulness check. This pipeline enables the creation of fine-grained hard negatives to probe text embedding models, revealing weaknesses in handling phenomena like negation and role shifts. By combining transparent symbolic edits with neural fluency and a validation step, the authors produced a dynamic benchmark that complements static datasets like MTEB and highlights model weaknesses across linguistic manipulations. The work demonstrates significant potential for targeted, interpretable evaluation of semantic understanding in embeddings and suggests avenues for multilingual extension and broader reflection on benchmarking practices.

Abstract

Controllable and transparent text generation has been a long-standing goal in NLP. Almost as long-standing is a general idea for addressing this challenge: Parsing text to a symbolic representation, and generating from it. However, earlier approaches were hindered by parsing and generation insufficiencies. Using modern parsers and a safety supervision mechanism, we show how close current methods come to this goal. Concretely, we propose the Sentence Smith framework for English, which has three steps: 1. Parsing a sentence into a semantic graph. 2. Applying human-designed semantic manipulation rules. 3. Generating text from the manipulated graph. A final entailment check (4.) verifies the validity of the applied transformation. To demonstrate our framework's utility, we use it to induce hard negative text pairs that challenge text embedding models. Since the controllable generation makes it possible to clearly isolate different types of semantic shifts, we can evaluate text embedding models in a fine-grained way, also addressing an issue in current benchmarking where linguistic phenomena remain opaque. Human validation confirms that our transparent generation process produces texts of good quality. Notably, our way of generation is very resource-efficient, since it relies only on smaller neural networks.

Figures (5)

• Figure 1: Example application of two controlled text transformations. Left: Switching semantic roles in a sentence, where the agent (ARG0: the biter) becomes the patient (ARG1: the bitten), and vice versa. Right: Deleting the patient and negating the main predicate.
• Figure 2: Breaking the paraphrase relation and creating a challenging test case: The new sentence has a high surface similarity to the input, but it is not a paraphrase. By contrast, the actual paraphrases have a lower superficial overlap, thus posing a challenge to any model, especially those that tend to rely on surface cues.
• Figure 3: Five aspectual semantic manipulations.
• Figure 4: Fine-grained linguistic benchmarking of text embedding models.
• Figure 5: Running example of the full data generation process with SentenceSmith. Under the strict quality filtering criterion of $>$ 90% contradiction, only PN and AR are retained from the five candidate generations. In the ablation study with the alternative, less stringent neutral filter, only the RS is retained.