Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

TL;DR

This work empirically evaluates LLMs as backbone encoders for dense retrieval across six capabilities, comparing more than 15 backbones from 0.1B to 32B parameters and including non-LLMs. It demonstrates that larger, well-pretrained LLMs consistently boost in-domain accuracy, data efficiency, and zero-shot and lengthy retrieval generalization, while instruction-based retrieval benefits LLMs but not non-LLMs. The results highlight the value of LLMs as versatile dense retriever backbones and offer practical guidance on configuring model size, pretraining sufficiency, and alignment to balance performance across tasks. Overall, the findings suggest that scaling and adequate pretraining of LLM backbones substantially elevate dense retrieval performance and adaptability in real-world scenarios.

Abstract

Pretrained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. However, these models often exhibit limited generalization capabilities and face challenges in improving in domain accuracy. Recent research has explored using large language models (LLMs) as retrievers, achieving SOTA performance across various tasks. Despite these advancements, the specific benefits of LLMs over traditional retrievers and the impact of different LLM configurations, such as parameter sizes, pretraining duration, and alignment processes on retrieval tasks remain unclear. In this work, we conduct a comprehensive empirical study on a wide range of retrieval tasks, including in domain accuracy, data efficiency, zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. We evaluate over 15 different backbone LLMs and non LLMs. Our findings reveal that larger models and extensive pretraining consistently enhance in domain accuracy and data efficiency. Additionally, larger models demonstrate significant potential in zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. These results underscore the advantages of LLMs as versatile and effective backbone encoders in dense retrieval, providing valuable insights for future research and development in this field.

Figures (5)

• Figure 1: In-domain accuracy (measured by MRR@10)
• Figure 2: Data efficiency
• Figure 3: Lengthy retrieval
• Figure 4: Zero-shot performance (measured by NDCG@10)
• Figure 5: Instrctions used in instruction-based retrieval.