Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieval-in-the-Chain: Bootstrapping Large Language Models for Generative Retrieval

Yingchen Zhang, Ruqing Zhang, Jiafeng Guo, Wenjun Peng, Sen Li, Fuyu Lv

TL;DR

This work addresses the gap between reasoning and retrieval in generative retrieval (GR) by introducing Reason-for-Retrieval (R4R), a framework that converts free-form chain-of-thought into concise, structured signals and iteratively refines retrieval. R4R leverages a single, instruction-tuned LLM to both reason and retrieve, using a Think-Retrieve-Refine loop with verification and reflection to progressively improve candidate docids. The approach is designed to preserve the model's natural language generation capabilities by using textual docids and retrieval-specific prompts, and it requires no additional models or training beyond instruction-tuning. Empirical results on Natural Questions, MS MARCO, and a Taobao item-search benchmark show consistent retrieval gains across multiple GR baselines with minimal decoding overhead, highlighting practical impact for real-world, large-scale retrieval tasks.

Abstract

Generative retrieval (GR) is an emerging paradigm that leverages large language models (LLMs) to autoregressively generate document identifiers (docids) relevant to a given query. Prior works have focused on leveraging the generative capabilities of LLMs to improve GR, while overlooking that their reasoning capabilities could likewise help. This raises a key question: Can explicit reasoning benefit GR? To investigate, we first conduct a preliminary study where an LLM is prompted to generate free-form chain-of-thought (CoT) reasoning before performing constrained docid decoding. Although this method outperforms standard GR, the generated reasoning tends to be verbose and poorly aligned with the docid space. These limitations motivate the development of a reasoning mechanism better tailored to GR. Therefore, we propose Reason-for-Retrieval (R4R), a reasoning-augmented framework for GR that converts free-form CoT reasoning into a compact, structured format, and iteratively refines the reasoning during the retrieval process. R4R augments an existing GR method by leveraging a reasoning-capable LLM that has been instruction-tuned for GR. At inference time, R4R first uses the LLM to generate an initial structured reasoning; then the same LLM alternates between (i) constrained decoding with the chosen GR method to produce candidate docids and (ii) updating the reasoning based on retrieval results to improve the next round. R4R does not require additional models or training, and instead a single LLM serves as both the reasoning generator and the retriever. Extensive experiments on Natural Questions, MS MARCO, and a real-world item-search benchmark validate the effectiveness of R4R.

Retrieval-in-the-Chain: Bootstrapping Large Language Models for Generative Retrieval

TL;DR

This work addresses the gap between reasoning and retrieval in generative retrieval (GR) by introducing Reason-for-Retrieval (R4R), a framework that converts free-form chain-of-thought into concise, structured signals and iteratively refines retrieval. R4R leverages a single, instruction-tuned LLM to both reason and retrieve, using a Think-Retrieve-Refine loop with verification and reflection to progressively improve candidate docids. The approach is designed to preserve the model's natural language generation capabilities by using textual docids and retrieval-specific prompts, and it requires no additional models or training beyond instruction-tuning. Empirical results on Natural Questions, MS MARCO, and a Taobao item-search benchmark show consistent retrieval gains across multiple GR baselines with minimal decoding overhead, highlighting practical impact for real-world, large-scale retrieval tasks.

Abstract

Generative retrieval (GR) is an emerging paradigm that leverages large language models (LLMs) to autoregressively generate document identifiers (docids) relevant to a given query. Prior works have focused on leveraging the generative capabilities of LLMs to improve GR, while overlooking that their reasoning capabilities could likewise help. This raises a key question: Can explicit reasoning benefit GR? To investigate, we first conduct a preliminary study where an LLM is prompted to generate free-form chain-of-thought (CoT) reasoning before performing constrained docid decoding. Although this method outperforms standard GR, the generated reasoning tends to be verbose and poorly aligned with the docid space. These limitations motivate the development of a reasoning mechanism better tailored to GR. Therefore, we propose Reason-for-Retrieval (R4R), a reasoning-augmented framework for GR that converts free-form CoT reasoning into a compact, structured format, and iteratively refines the reasoning during the retrieval process. R4R augments an existing GR method by leveraging a reasoning-capable LLM that has been instruction-tuned for GR. At inference time, R4R first uses the LLM to generate an initial structured reasoning; then the same LLM alternates between (i) constrained decoding with the chosen GR method to produce candidate docids and (ii) updating the reasoning based on retrieval results to improve the next round. R4R does not require additional models or training, and instead a single LLM serves as both the reasoning generator and the retriever. Extensive experiments on Natural Questions, MS MARCO, and a real-world item-search benchmark validate the effectiveness of R4R.
Paper Structure (22 sections, 9 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 22 sections, 9 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Exploration and challenges
  3. Standard GR
  4. Adapting GR for reasoning
  5. Direct CoT for GR
  6. Method
  7. Overview
  8. Think: initial reasoning generation
  9. Iterating Retrieve-Refine
  10. Retrieve
  11. Refine
  12. Iterating retrieve and refine
  13. Integration with existing GR methods
  14. Experimental setup
  15. Experimental results
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Comparison of (a) standard GR, (b) GR + Direct CoT, and (c) our proposed R4R. R4R compresses and structured reasoning, forming an iterative improvement pipeline
  • Figure 2: Illustration of the initial Think step and the Retrieve-Refine iteration in R4R.
  • Figure 3: Performance trends of R4R as the verify depth varies on NQ. The left plot shows R4R integrating DSI and the right plot shows R4R integrating SEAL.
  • Figure 4: Performance and efficiency trends of DSI+R4R as the round budget varies on NQ.