Listwise Generative Retrieval Models via a Sequential Learning Process

TL;DR

This article views the generation of a ranked docid list as a sequence learning process, which empowers the GR model to optimize the relevance at the docid list level, and designs a positional conditional probability for GR.

Abstract

Recently, a novel generative retrieval (GR) paradigm has been proposed, where a single sequence-to-sequence model is learned to directly generate a list of relevant document identifiers (docids) given a query. Existing GR models commonly employ maximum likelihood estimation (MLE) for optimization: this involves maximizing the likelihood of a single relevant docid given an input query, with the assumption that the likelihood for each docid is independent of the other docids in the list. We refer to these models as the pointwise approach in this paper. While the pointwise approach has been shown to be effective in the context of GR, it is considered sub-optimal due to its disregard for the fundamental principle that ranking involves making predictions about lists. In this paper, we address this limitation by introducing an alternative listwise approach, which empowers the GR model to optimize the relevance at the docid list level. Specifically, we view the generation of a ranked docid list as a sequence learning process: at each step we learn a subset of parameters that maximizes the corresponding generation likelihood of the $i$-th docid given the (preceding) top $i-1$ docids. To formalize the sequence learning process, we design a positional conditional probability for GR. To alleviate the potential impact of beam search on the generation quality during inference, we perform relevance calibration on the generation likelihood of model-generated docids according to relevance grades. We conduct extensive experiments on representative binary and multi-graded relevance datasets. Our empirical results demonstrate that our method outperforms state-of-the-art GR baselines in terms of retrieval performance.

Figures (5)

• Figure 1: Optimization objectives. Assume that the following are given: a query $q$ and two ground-truth docids, $docid_1$ and $docid_2$, where $docid_1$ is more relevant than $docid_2$ to $q$. Top: Most existing GR work relies on maximum likelihood estimation, by maximizing the likelihood of the target docid for each query-docid pair. All relevant docids $docid_1$ and $docid_2$ are treated equally, sharing similar likelihood values. Bottom: A listwise objective (yellow rectangle) is designed for GR, directly modeling the ranked docid lists and incorporating positional information between $docid_1$ and $docid_2$ ($docid_1$ with darker green has a larger positional weight), resulting in a positional weighted likelihood.
• Figure 2: Overview of the two-stage listwise learning methods, which consists of a training stage using listwise loss and a re-training stage with relevance calibration based on the trained model.
• Figure 3: Training with limited supervision data. The x-axis indicates the number of training queries.
• Figure 4: During the training stage, different numbers of relevance grades in the ClueWeb 200K dataset are used in the listwise loss (Eq. \ref{['eq:listwise-GR']}). The x-axis represents the number of relevance grades used, indicated in parentheses as the combination of the relevance grades or the corresponding model names.
• Figure 5: t-SNE plots of query and document representations for ListGR and NCI. The representations are the output of the encoder of ListGR and NCI.