Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Retrieval: End-to-End Information Retrieval with One Large Language Model

Qiaoyu Tang, Jiawei Chen, Zhuoqun Li, Bowen Yu, Yaojie Lu, Cheng Fu, Haiyang Yu, Hongyu Lin, Fei Huang, Ben He, Xianpei Han, Le Sun, Yongbin Li

TL;DR

Self-Retrieval proposes an end-to-end IR system powered by a single large language model that internalizes the corpus, generates relevant passages with constrained decoding, and performs self-assessment for reranking. By unifying indexing, retrieval, and reranking within one model, it enables seamless information sharing across IR components and reduces error propagation inherent in pipelines. Empirical results on NQ, TriviaQA, and MS MARCO show substantial gains over sparse, dense, and generative baselines, with clear scaling benefits for larger LLMs and corpora. The approach also demonstrates strong performance in retrieval-augmented generation, highlighting practical impact for real-world RAG applications.

Abstract

The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them. In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture. Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process. Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning, transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking. Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation.

Self-Retrieval: End-to-End Information Retrieval with One Large Language Model

TL;DR

Self-Retrieval proposes an end-to-end IR system powered by a single large language model that internalizes the corpus, generates relevant passages with constrained decoding, and performs self-assessment for reranking. By unifying indexing, retrieval, and reranking within one model, it enables seamless information sharing across IR components and reduces error propagation inherent in pipelines. Empirical results on NQ, TriviaQA, and MS MARCO show substantial gains over sparse, dense, and generative baselines, with clear scaling benefits for larger LLMs and corpora. The approach also demonstrates strong performance in retrieval-augmented generation, highlighting practical impact for real-world RAG applications.

Abstract

The rise of large language models (LLMs) has significantly transformed both the construction and application of information retrieval (IR) systems. However, current interactions between IR systems and LLMs remain limited, with LLMs merely serving as part of components within IR systems, and IR systems being constructed independently of LLMs. This separated architecture restricts knowledge sharing and deep collaboration between them. In this paper, we introduce Self-Retrieval, a novel end-to-end LLM-driven information retrieval architecture. Self-Retrieval unifies all essential IR functions within a single LLM, leveraging the inherent capabilities of LLMs throughout the IR process. Specifically, Self-Retrieval internalizes the retrieval corpus through self-supervised learning, transforms the retrieval process into sequential passage generation, and performs relevance assessment for reranking. Experimental results demonstrate that Self-Retrieval not only outperforms existing retrieval approaches by a significant margin, but also substantially enhances the performance of LLM-driven downstream applications like retrieval-augmented generation.
Paper Structure (31 sections, 3 equations, 4 figures, 9 tables)

This paper contains 31 sections, 3 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM for IR
  4. Generative retrieval
  5. Self-Retrieval
  6. Indexing: Internalize the Corpus
  7. Retrieval: Generate Relevant Passage through Constrained Decoding
  8. Reranking: Assess the Relevance
  9. Training & Inference
  10. Training
  11. Inference
  12. Experimental Results
  13. Experimental Setup
  14. Datasets and metrics
  15. Implementation details
  16. ...and 16 more sections

Figures (4)

  • Figure 1: The Self-Retrieval framework consists of three key components: (1) corpus indexing through self-supervised learning, (2) passage generation via constrained decoding, (3) passage ranking using self-assessment scoring.
  • Figure 2: Impact of model capacity on Self-Retrieval performance.
  • Figure 3: Reranking performance comparison when processing top-100 passages.
  • Figure 4: Scalability analysis of retrieval performance for Self-Retrieval and BGE-FT across varying corpus sizes.