Large Language Models are Strong Zero-Shot Retriever

TL;DR

LameR reframes zero-shot large-scale retrieval by using a frozen LLM to generate candidate-driven query augmentations while relying on a fast, non-parametric BM25 retriever. By prompting the LLM with top-M retrieved candidates, it elicits multiple plausible answers that guide the final retrieval through concatenation-based augmentation, preserving efficiency and transparency. Across MS-MARCO DL19/DL20 and BEIR benchmarks, LameR achieves strong zero-shot performance, often surpassing HyDE-based dense approaches and matching or exceeding supervised baselines in several tasks. The approach highlights the practical value of integrating in-domain demonstrations via retrieved candidates and confirms that lexicon-based retrieval can effectively leverage LLM outputs without heavy neural retrievers. Limitations include prompt sensitivity and LLM inference overhead, suggesting future work with smaller models or more robust prompting strategies.

Abstract

In this work, we propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM, while breaking brute-force combinations of retrievers with LLMs and lifting the performance of zero-shot retrieval to be very competitive on benchmark datasets. Essentially, we propose to augment a query with its potential answers by prompting LLMs with a composition of the query and the query's in-domain candidates. The candidates, regardless of correct or wrong, are obtained by a vanilla retrieval procedure on the target collection. As a part of the prompts, they are likely to help LLM generate more precise answers by pattern imitation or candidate summarization. Even if all the candidates are wrong, the prompts at least make LLM aware of in-collection patterns and genres. Moreover, due to the low performance of a self-supervised retriever, the LLM-based query augmentation becomes less effective as the retriever bottlenecks the whole pipeline. Therefore, we propose to leverage a non-parametric lexicon-based method (e.g., BM25) as the retrieval module to capture query-document overlap in a literal fashion. As such, LameR makes the retrieval procedure transparent to the LLM, thus circumventing the performance bottleneck.

Figures (5)

• Figure 1: nDCG@10 on DL19 for query augmentation w/ LLMs.
• Figure 2: HyDE improving Dense and Term-based Retrieval.
• Figure 3: Large language model as Retriever (LameR). Please see Table \ref{['tab:prompt_example']} for the prompt formulation.
• Figure 4: Hyperparameter explorations and ablation studies, where the data points in dashed rectangles denote our default choices. (a) The number of retrieved passages as in-context demonstration for answer generation, i.e., $M$ in Eq.(\ref{['equ:candi_gen']}). (b) The number of generated answers as query augmentations for large-scale retrieval, i.e., $N$ in Eq.(\ref{['equ:ans_gen']}). (c) and (d) depict the schemes to obtain the 10 demo-passages, where the first is to fetch 10 consecutive passages from a start index of the BM25-retrieved passages and the second is to randomly sample 10 passages from top-N passages. Note that '$\gg$1k' denotes randomly sampling 10 passages from the whole collection.
• Figure 5: Efficiency of LameR with HyDE in retrieval latency (QPS) and index size (GB). Numbers for LameR sum overheads in two stages, and the variants for each system are achieved by changing generation number.