Enhancing Lexicon-Based Text Embeddings with Large Language Models

TL;DR

This work tackles tokenization redundancy and unidirectional attention in lexicon-based text embeddings by introducing LENS, a framework that clusters LM token embeddings into $k$ centroids and enables bidirectional attention. LENS produces compact, $k$-dimensional lexicon embeddings that outperform dense baselines on the MTEB benchmark and, when fused with dense embeddings, achieve state-of-the-art results on BEIR retrieval. The approach relies on a simple, public-data–driven training regime with InfoNCE loss and LoRA, preserving transparency through token-cluster associations. Overall, LENS demonstrates strong generalization, robust clustering behavior, and complementary gains when combined with dense representations, albeit with higher computational cost and English-only evaluation in this study.

Abstract

Recent large language models (LLMs) have demonstrated exceptional performance on general-purpose text embedding tasks. While dense embeddings have dominated related research, we introduce the first Lexicon-based EmbeddiNgS (LENS) leveraging LLMs that achieve competitive performance on these tasks. Regarding the inherent tokenization redundancy issue and unidirectional attention limitations in traditional causal LLMs, LENS consolidates the vocabulary space through token embedding clustering, and investigates bidirectional attention and various pooling strategies. Specifically, LENS simplifies lexicon matching by assigning each dimension to a specific token cluster, where semantically similar tokens are grouped together, and unlocking the full potential of LLMs through bidirectional attention. Extensive experiments demonstrate that LENS outperforms dense embeddings on the Massive Text Embedding Benchmark (MTEB), delivering compact feature representations that match the sizes of dense counterparts. Notably, combining LENSE with dense embeddings achieves state-of-the-art performance on the retrieval subset of MTEB (i.e. BEIR).

Figures (3)

• Figure 1: The redundancy and noise in LLM tokenizers, as well as the absence of bidirectional dependencies in causal LLMs motivate LENS.
• Figure 2: The model framework of LENS.
• Figure 3: Influence of the number of clusters. The configuration with 32,000 clusters retains the original token embeddings without clustering.