Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval

TL;DR

BootRet presents a bootstrapped pre-training framework for generative retrieval where document identifiers are dynamically updated to match evolving model parameters. It introduces two pre-training tasks (corpus indexing and relevance prediction) augmented with noisy documents and pseudo-queries generated by large language models, plus contrastive and semantic-consistency losses. An enhanced bootstrapping loop alternates updating model parameters and docids, yielding substantial improvements over strong baselines and showing notable zero-shot and low-resource advantages. The approach demonstrates that dynamically aligning docids with the learning trajectory can significantly boost retrieval effectiveness and efficiency in generative retrieval settings.

Abstract

Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query. Recent studies have highlighted the potential of a strong generative retrieval model, trained with carefully crafted pre-training tasks, to enhance downstream retrieval tasks via fine-tuning. However, the full power of pre-training for generative retrieval remains underexploited due to its reliance on pre-defined static document identifiers, which may not align with evolving model parameters. In this work, we introduce BootRet, a bootstrapped pre-training method for generative retrieval that dynamically adjusts document identifiers during pre-training to accommodate the continuing memorization of the corpus. BootRet involves three key training phases: (i) initial identifier generation, (ii) pre-training via corpus indexing and relevance prediction tasks, and (iii) bootstrapping for identifier updates. To facilitate the pre-training phase, we further introduce noisy documents and pseudo-queries, generated by large language models, to resemble semantic connections in both indexing and retrieval tasks. Experimental results demonstrate that BootRet significantly outperforms existing pre-training generative retrieval baselines and performs well even in zero-shot settings.

Figures (6)

• Figure 1: The bootstrapped pre-training pipeline of BootRet. (1) The initial docids $\mathcal{I}_\mathcal{D}^0$ are obtained with the initial model parameters $\theta^0$. (2) To perform the $t$-th iteration, we design the corpus indexing task and relevance prediction task for pre-training. We construct noisy documents and pseudo-queries with a LLM, and design contrastive losses (the yellow and the orange rectangles) and a semantic consistency loss (the green rectangle) to learn the corpus and relevance information discriminatively. After pre-training, the model updates from $\theta^{t-1}$ to $\theta^{t}$. (3) The bootstrapped $\theta^{t}$ is used to dynamically update the docids $\mathcal{I}_\mathcal{D}^{t-1}$ to $\mathcal{I}_\mathcal{D}^{t}$, i.e., bootstrapped docids, which are further used in the next iteration. (Figure should be viewed in color.)
• Figure 2: Results under zero- and low-resource setting. The x-axis indicates the number of labeled queries.
• Figure 3: Retrieval performance of different number of iterations on MS 300K.
• Figure 4: Retrieval performance of different number of iterations on NQ.
• Figure 5: Retrieval performance of different prompts for generating noisy documents during pre-training.