Generative Retrieval Meets Multi-Graded Relevance

TL;DR

This work creates identifiers that are both semantically relevant and sufficiently distinct to represent individual documents effectively through a combination of docid generation and autoencoder models, and implements multi-graded constrained contrastive training.

Abstract

Generative retrieval represents a novel approach to information retrieval. It uses an encoder-decoder architecture to directly produce relevant document identifiers (docids) for queries. While this method offers benefits, current approaches are limited to scenarios with binary relevance data, overlooking the potential for documents to have multi-graded relevance. Extending generative retrieval to accommodate multi-graded relevance poses challenges, including the need to reconcile likelihood probabilities for docid pairs and the possibility of multiple relevant documents sharing the same identifier. To address these challenges, we introduce a framework called GRaded Generative Retrieval (GR$^2$). GR$^2$ focuses on two key components: ensuring relevant and distinct identifiers, and implementing multi-graded constrained contrastive training. First, we create identifiers that are both semantically relevant and sufficiently distinct to represent individual documents effectively. This is achieved by jointly optimizing the relevance and distinctness of docids through a combination of docid generation and autoencoder models. Second, we incorporate information about the relationship between relevance grades to guide the training process. We use a constrained contrastive training strategy to bring the representations of queries and the identifiers of their relevant documents closer together, based on their respective relevance grades. Extensive experiments on datasets with both multi-graded and binary relevance demonstrate the effectiveness of GR$^2$.

Figures (5)

• Figure 1: A Seq2Seq encoder-decoder architecture is used to consume queries and produce relevant docids for GR. We employ a multi-graded constrained contrastive loss (Section \ref{['sec:con']}) to characterize the relationships among relevance labels based on the relevant and distinct docids (Section \ref{['sec:Docid']}).
• Figure 2: Ablation analysis. (Left) Supervised learning; (Right) Pre-training and fine-tuning.
• Figure 3: Supervised training and fine-tuning with limited supervision data. The x-axis indicates the number of training queries.
• Figure 4: The regularized fusion approach to generate relevant and distinct docids.
• Figure 5: t-SNE plots of query and document representations for GR$^{2\mathit{P}}$ (left), RIPOR (mid) and NCI (right).