Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewriting

TL;DR

This work tackles embedding performance limitations by using large language models to enrich and rewrite input text before embedding. The authors leverage ChatGPT 3.5 with four prompt variations to perform context enrichment, grammatical correction, terminology normalization, disambiguation, acronym expansion, metadata incorporation, sentence restructuring, and missing-information inference, evaluating with a text-embedding-3-large baseline across Banking77Classification, TwitterSemEval 2015, and Amazon Counter Factual Classification. Numerical results show substantial gains on TwitterSemEval 2015—up to a cosine similarity of 85.34 on the MTEB Leaderboard with the best prompt—while improvements on Banking77Classification and Amazon Counter Factual Classification are less pronounced. The findings indicate that LLM-based text enrichment can meaningfully enhance embedding quality in domain-sensitive settings, though cross-domain variability highlights the need for tailored prompt design and domain-specific considerations.

Abstract

Embedding models are crucial for various natural language processing tasks but can be limited by factors such as limited vocabulary, lack of context, and grammatical errors. This paper proposes a novel approach to improve embedding performance by leveraging large language models (LLMs) to enrich and rewrite input text before the embedding process. By utilizing ChatGPT 3.5 to provide additional context, correct inaccuracies, and incorporate metadata, the proposed method aims to enhance the utility and accuracy of embedding models. The effectiveness of this approach is evaluated on three datasets: Banking77Classification, TwitterSemEval 2015, and Amazon Counter-factual Classification. Results demonstrate significant improvements over the baseline model on the TwitterSemEval 2015 dataset, with the best-performing prompt achieving a score of 85.34 compared to the previous best of 81.52 on the Massive Text Embedding Benchmark (MTEB) Leaderboard. However, performance on the other two datasets was less impressive, highlighting the importance of considering domain-specific characteristics. The findings suggest that LLM-based text enrichment has shown promising results to improve embedding performance, particularly in certain domains. Hence, numerous limitations in the process of embedding can be avoided.