LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference

TL;DR

LightRetriever introduces an asymmetric LLM-based retrieval framework that keeps a full document encoder while replacing the query encoder with a lightweight, cacheable embedding lookup. By training dense and sparse components with instruction-guided prompts and using caching to precompute token embeddings, it achieves extreme online query efficiency (up to >1000x faster query encoding) and substantial end-to-end throughput gains (>$10x$) without large losses in retrieval performance (averaging around 95% of full symmetric baselines). The approach is validated on BeIR and CMTEB benchmarks across multiple LLM backbones, demonstrating broad generalization and practical viability for latency-critical retrieval tasks. Ablation studies show the necessity of deep document-side modeling, the value of hybrid dense+sparse signals, and controllable index/embedding sizes via Matryoshka representations, enabling deployment in resource-constrained environments.

Abstract

Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded offline, while queries arrive in real-time, necessitating an efficient online query encoder. Although LLMs significantly enhance retrieval capabilities, serving deeply parameterized LLMs slows down query inference throughput and increases demands for online deployment resources. In this paper, we propose LightRetriever, a novel LLM-based retriever with extremely lightweight query encoders. Our method retains a full-sized LLM for document encoding, but reduces the workload of query encoding to no more than an embedding lookup. Compared to serving a full LLM on an A800 GPU, our method achieves over 1000x speedup in query encoding and over 10x increase in end-to-end retrieval throughput. Extensive experiments on large-scale retrieval benchmarks show that LightRetriever generalizes well across diverse tasks, maintaining an average of 95% retrieval performance.

Figures (7)

• Figure 1: LightRetriever targets at extreme query inference speed for LLM-based text retrieval, reducing the workload of query encoding to no more than an embedding lookup.
• Figure 2: Contrastive training of full-sized symmetric dense retrieval. A full-sized dual-encoder is used to model both queries and documents. Task-specific instructions are added as common practices Su2023InstructorWang2024mistral-E5 to promote better domain adaptation abilities.
• Figure 3: Dense retrieval of LightRetriever. Three stages are involved for efficient query modeling. 1) Training: A shared prompt concatenated with a single query token is passed through a full encoder independently to obtain token-level representations. The full query representation is then computed by averaging the token vectors corresponding to each query token. 2) Caching: Prior to serving, the trained encoder is used to precompute token embeddings for the entire LLM's vocabulary, which are stored in a single embedding lookup matrix. 3) Online Serving: At inference time, query embeddings are efficiently generated by lookup and averaging the cached token embeddings, eliminating the need for deep model inference.
• Figure 4: Sparse retrieval of LightRetriever. The query representation is a term-based, unlearnable sparse vector, which is obtained from tokenization. The document representation is end-to-end trained by projecting last layer hidden states of LLM to vocabulary space, then optimized via contrastive learning.
• Figure 5: Ablations of retrieval performances (nDCG@10) of LightRetriever on BeIR and CMTEB-R benchmarks. For detailed results, please refer to Table \ref{['table:beir_detail_part1']}, \ref{['table:beir_detail_part2']} and \ref{['table:cmteb_detail']} in Appendix.