Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Constrained Auto-Regressive Decoding Constrains Generative Retrieval

Shiguang Wu, Zhaochun Ren, Xin Xin, Jiyuan Yang, Mengqi Zhang, Zhumin Chen, Maarten de Rijke, Pengjie Ren

TL;DR

This paper analyzes constrained auto-regressive decoding in generative retrieval under a Bayes-optimal model, identifying fundamental limits from corpus-level constraints and decoding strategies. It derives a KL-divergence based lower bound on the error caused by constraint-induced marginal mismatches and demonstrates that beam search may degrade recall when using marginal distributions. Through synthetic and real-data experiments, the authors corroborate the theoretical findings and reveal a tension between concentration of the relevance distribution and reliable constraint satisfaction. The work provides a theoretical foundation for the limitations of generative retrieval and motivates future design of robust, generalizable decoding strategies.

Abstract

Generative retrieval seeks to replace traditional search index data structures with a single large-scale neural network, offering the potential for improved efficiency and seamless integration with generative large language models. As an end-to-end paradigm, generative retrieval adopts a learned differentiable search index to conduct retrieval by directly generating document identifiers through corpus-specific constrained decoding. The generalization capabilities of generative retrieval on out-of-distribution corpora have gathered significant attention. In this paper, we examine the inherent limitations of constrained auto-regressive generation from two essential perspectives: constraints and beam search. We begin with the Bayes-optimal setting where the generative retrieval model exactly captures the underlying relevance distribution of all possible documents. Then we apply the model to specific corpora by simply adding corpus-specific constraints. Our main findings are two-fold: (i) For the effect of constraints, we derive a lower bound of the error, in terms of the KL divergence between the ground-truth and the model-predicted step-wise marginal distributions. (ii) For the beam search algorithm used during generation, we reveal that the usage of marginal distributions may not be an ideal approach. This paper aims to improve our theoretical understanding of the generalization capabilities of the auto-regressive decoding retrieval paradigm, laying a foundation for its limitations and inspiring future advancements toward more robust and generalizable generative retrieval.

Constrained Auto-Regressive Decoding Constrains Generative Retrieval

TL;DR

This paper analyzes constrained auto-regressive decoding in generative retrieval under a Bayes-optimal model, identifying fundamental limits from corpus-level constraints and decoding strategies. It derives a KL-divergence based lower bound on the error caused by constraint-induced marginal mismatches and demonstrates that beam search may degrade recall when using marginal distributions. Through synthetic and real-data experiments, the authors corroborate the theoretical findings and reveal a tension between concentration of the relevance distribution and reliable constraint satisfaction. The work provides a theoretical foundation for the limitations of generative retrieval and motivates future design of robust, generalizable decoding strategies.

Abstract

Generative retrieval seeks to replace traditional search index data structures with a single large-scale neural network, offering the potential for improved efficiency and seamless integration with generative large language models. As an end-to-end paradigm, generative retrieval adopts a learned differentiable search index to conduct retrieval by directly generating document identifiers through corpus-specific constrained decoding. The generalization capabilities of generative retrieval on out-of-distribution corpora have gathered significant attention. In this paper, we examine the inherent limitations of constrained auto-regressive generation from two essential perspectives: constraints and beam search. We begin with the Bayes-optimal setting where the generative retrieval model exactly captures the underlying relevance distribution of all possible documents. Then we apply the model to specific corpora by simply adding corpus-specific constraints. Our main findings are two-fold: (i) For the effect of constraints, we derive a lower bound of the error, in terms of the KL divergence between the ground-truth and the model-predicted step-wise marginal distributions. (ii) For the beam search algorithm used during generation, we reveal that the usage of marginal distributions may not be an ideal approach. This paper aims to improve our theoretical understanding of the generalization capabilities of the auto-regressive decoding retrieval paradigm, laying a foundation for its limitations and inspiring future advancements toward more robust and generalizable generative retrieval.

Paper Structure

This paper contains 24 sections, 4 theorems, 17 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Theoretical analysis
  5. Constraints cause marginal distribution mismatch
  6. The impact of beam search on recall
  7. Concentration as a trade-off factor
  8. Difficulty of concentration in Bayes-optimal gr
  9. Aggregation introduces redundancy
  10. Amplification requires high-quality data
  11. Synthetic experiments
  12. Effects of constraints
  13. Effects of beam search
  14. Summary
  15. Experiments on real-world dataset
  16. ...and 9 more sections

Key Result

Lemma B.1

Let $\boldsymbol{S}_i\sim \mathsf{Binomial}(m,p)$, where $i\in[k]$, and $Z=\sum_{i=1}^k S_i$. We define a normalized distribution $P$ as and a uniform distribution $Q$ on $\mathsf{supp}(P)\coloneqq \{i|S_i>0\}$ as Then, we have a lower bound of the kl divergence between $P$ and $Q$ for large $k$,

Figures (6)

  • Figure 1: The KL divergence error in the first generation step on synthetic uniform relevance distribution data with uniformly sampled downstream corpus.
  • Figure 2: The KL divergence error in the first generation step on several synthetic relevance distribution data with different degrees of concentration. The vocabulary size is $2^{10}$, the docid length is $3$. $A$ is the Simpson diversity index of the relevance distribution. As for highly concentrated distributions, e.g., $A=1$, and $A=0.04$, the Lyapunov's condition will no longer hold (see Theorem \ref{['the:kl_arbitrary']} for more details), we do not draw their lower bounds.
  • Figure 3: The recall of relevant branches cut off at the total number of relevant branches in the first generation step. The synthetic relevance distribution is constructed as Section \ref{['sub:overtaken']}. The total number of sampled relevant document is $\lambda k$. The size of each branch is fixed as $n=2^{25}$.
  • Figure 4: The recall of relevant branches cut off at the total number of relevant branches in the first generation step. The synthetic relevance distribution is constructed as Section \ref{['sub:overtaken']}. The total number of sampled relevant document is $0.05 k$. The size of each branch is fixed as $n=2^{25}$.
  • Figure 5: The KL divergence error in the first generation step on MS MARCO V1 passage corpus. A subset of top-$10,000$ rank list from SLIM++ liSLIM2023 is treated as the complete corpus, and the downstream corpus is (a) uniformly sampled, or (b) sampled with the relevance distribution.
  • ...and 1 more figures

Theorems & Definitions (5)

  • Lemma B.1: Lower bound of kl divergence between Binomial and uniform distribution
  • Theorem B.2: Lower bound of kl divergence for uniform relevance distribution.
  • Theorem B.3: Lower bound of kl divergence for general relevance distribution
  • Theorem C.1: Top-$\lambda k$ recall and top-$1$ precision of data model in Section \ref{['sub:overtaken']}
  • Remark C.2