When Text Embedding Meets Large Language Model: A Comprehensive Survey
Zhijie Nie, Zhangchi Feng, Mingxin Li, Cunwang Zhang, Yanzhao Zhang, Dingkun Long, Richong Zhang
TL;DR
This survey addresses how large language models (LLMs) interact with and transform text embeddings. It introduces a three-part taxonomy: LLM-augmented text embedding, LLMs as text embedders, and embedding understanding with LLMs, and situates this within a historical trajectory from statistic methods to pre-trained language models. It highlights two new downstream tasks—long context compression and embedding inversion—and reviews a broad spectrum of data-synthesis, annotation, architectural, and optimization techniques enabling LLM-enhanced embeddings. The findings show that LLMs substantially reduce data-annotation needs and bolster cross-task generalization, while also uncovering new challenges such as privacy leakage, high-dimensional embeddings, and resource overhead that motivate further technical innovation. Collectively, the work guides researchers toward cross-lingual and cross-modal embedding solutions, task-specific encoders, and interpretable representations in the rapidly evolving NLP landscape.
Abstract
Text embedding has become a foundational technology in natural language processing (NLP) during the deep learning era, driving advancements across a wide array of downstream tasks. While many natural language understanding challenges can now be modeled using generative paradigms and leverage the robust generative and comprehension capabilities of large language models (LLMs), numerous practical applications - such as semantic matching, clustering, and information retrieval - continue to rely on text embeddings for their efficiency and effectiveness. Therefore, integrating LLMs with text embeddings has become a major research focus in recent years. In this survey, we categorize the interplay between LLMs and text embeddings into three overarching themes: (1) LLM-augmented text embedding, enhancing traditional embedding methods with LLMs; (2) LLMs as text embedders, adapting their innate capabilities for high-quality embedding; and (3) Text embedding understanding with LLMs, leveraging LLMs to analyze and interpret embeddings. By organizing recent works based on interaction patterns rather than specific downstream applications, we offer a novel and systematic overview of contributions from various research and application domains in the era of LLMs. Furthermore, we highlight the unresolved challenges that persisted in the pre-LLM era with pre-trained language models (PLMs) and explore the emerging obstacles brought forth by LLMs. Building on this analysis, we outline prospective directions for the evolution of text embedding, addressing both theoretical and practical opportunities in the rapidly advancing landscape of NLP.