PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs

TL;DR

PTEB introduces a dynamic evaluation protocol that generates eval-time paraphrases of MTEB data using LLMs to stress-test sentence embedding models for semantic invariance. By grounding paraphrase quality in STS gold ratings and selecting a capable LLM judge and paraphraser, the method produces a stochastic population of semantically equivalent yet token-diverse test instances. Across 7 MTEB tasks and 10 languages, embeddings generally degrade under paraphrase perturbations, revealing robustness gaps and showing that smaller models can be competitive under PTEB with appropriate configurations. The work proposes PTEB as a complementary, compute-bounded evaluation paradigm designed to reduce benchmark saturation and overfitting while enabling broader, reproducible robustness assessments in multilingual settings.

Abstract

Current evaluations of sentence embedding models typically rely on static test beds such as the Massive Text Embedding Benchmark (MTEB). While invaluable, repeated tuning on a fixed suite can inflate reported performance and obscure real-world robustness. We introduce the Paraphrasing Text Embedding Benchmark (PTEB), a dynamic protocol that stochastically generates meaning-preserving paraphrases at evaluation time and aggregates results across multiple runs. Using a cost-efficient LLM-based method grounded in semantic textual similarity gold ratings, we show that LLMs generate token-diverse but semantically preserving, paraphrases. Across 7 MTEB tasks, we validate our hypothesis that the performance of sentence encoders is sensitive to changes in token space even when semantics remain fixed. We also observe that smaller models are not disproportionately affected relative to larger ones. Our results are statistically robust over multiple runs and we extended our experiments to 3 multilingual datasets covering 10 languages. More generally, we aim to propose a new evaluation paradigm in NLP that relies less on static, pre-defined benchmarks but shifts towards dynamic, stochastic evaluation leveraging eval-time compute.

Figures (6)

• Figure 1: PTEB paraphrases data at evaluation time to measure embedding model performance.
• Figure 2: First, MTEB STS data is used to evaluate different generative LLMs for their STS rating capability. Then, using the selected LLM judge, generative LLMs are evaluated for paraphrasing MTEB STS data. Finally, the best paraphrase model is used to paraphrase MTEB datasets at eval-time generating a dynamic benchmark for embedding models.
• Figure 3: LLM judge mean average error (MAE) and number of examples per gold score on MTEB STS.
• Figure 4: LLM judge scores on MTEB STS for different word count intervals; empty cells indicate that the dataset does not contain sentences with the respective word count. (in %)
• Figure 5: STSB paraphrase examples.