Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval

Penghao Lu, Xin Dong, Yuansheng Zhou, Lei Cheng, Chuan Yuan, Linjian Mo

TL;DR

DOGR tackles the gap in generative retrieval where modeling only query–identifier relations misses direct query–document relevance. It introduces a two-stage framework that first learns document identifiers via keyword-based lexical representations and then refines document ranking through two contrastive losses derived from prefix-based and retrieval-augmented negatives. By fusing generation probabilities with semantic scores, DOGR achieves state-of-the-art results on NQ320k and MS MARCO and generalizes across common identifier constructions. The approach demonstrates that document-oriented contrastive learning enhances generative retrieval's understanding of query–document relevance, enabling robust performance in large-scale corpora. This method offers practical impact for systems seeking end-to-end generative retrieval without heavy reliance on external indexes.

Abstract

Generative retrieval constitutes an innovative approach in information retrieval, leveraging generative language models (LM) to generate a ranked list of document identifiers (docid) for a given query. It simplifies the retrieval pipeline by replacing the large external index with model parameters. However, existing works merely learned the relationship between queries and document identifiers, which is unable to directly represent the relevance between queries and documents. To address the above problem, we propose a novel and general generative retrieval framework, namely Leveraging Document-Oriented Contrastive Learning in Generative Retrieval (DOGR), which leverages contrastive learning to improve generative retrieval tasks. It adopts a two-stage learning strategy that captures the relationship between queries and documents comprehensively through direct interactions. Furthermore, negative sampling methods and corresponding contrastive learning objectives are implemented to enhance the learning of semantic representations, thereby promoting a thorough comprehension of the relationship between queries and documents. Experimental results demonstrate that DOGR achieves state-of-the-art performance compared to existing generative retrieval methods on two public benchmark datasets. Further experiments have shown that our framework is generally effective for common identifier construction techniques.

DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval

TL;DR

DOGR tackles the gap in generative retrieval where modeling only query–identifier relations misses direct query–document relevance. It introduces a two-stage framework that first learns document identifiers via keyword-based lexical representations and then refines document ranking through two contrastive losses derived from prefix-based and retrieval-augmented negatives. By fusing generation probabilities with semantic scores, DOGR achieves state-of-the-art results on NQ320k and MS MARCO and generalizes across common identifier constructions. The approach demonstrates that document-oriented contrastive learning enhances generative retrieval's understanding of query–document relevance, enabling robust performance in large-scale corpora. This method offers practical impact for systems seeking end-to-end generative retrieval without heavy reliance on external indexes.

Abstract

Generative retrieval constitutes an innovative approach in information retrieval, leveraging generative language models (LM) to generate a ranked list of document identifiers (docid) for a given query. It simplifies the retrieval pipeline by replacing the large external index with model parameters. However, existing works merely learned the relationship between queries and document identifiers, which is unable to directly represent the relevance between queries and documents. To address the above problem, we propose a novel and general generative retrieval framework, namely Leveraging Document-Oriented Contrastive Learning in Generative Retrieval (DOGR), which leverages contrastive learning to improve generative retrieval tasks. It adopts a two-stage learning strategy that captures the relationship between queries and documents comprehensively through direct interactions. Furthermore, negative sampling methods and corresponding contrastive learning objectives are implemented to enhance the learning of semantic representations, thereby promoting a thorough comprehension of the relationship between queries and documents. Experimental results demonstrate that DOGR achieves state-of-the-art performance compared to existing generative retrieval methods on two public benchmark datasets. Further experiments have shown that our framework is generally effective for common identifier construction techniques.

Paper Structure

This paper contains 25 sections, 7 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Document Retrieval
  4. Generative Retrieval
  5. The Proposed Scheme
  6. Identifier Generation Stage
  7. Keyword-Based Lexical Identifier.
  8. Pointwise Generation Task.
  9. Document Ranking Stage
  10. Prefix-oriented Negative Sampling.
  11. Retrieval-augmented Negative Sampling.
  12. Inference
  13. Experiments
  14. Datasets
  15. Metrics
  16. ...and 10 more sections

Figures (1)

  • Figure 1: An overview of the framework of DOGR. The training phase can be divided into two stages. The first is the identifier generation stage, which is used to learn the relationship between text and document identifiers. The second is the document ranking stage, which employs contrastive learning to learn semantic representation for document ranking. In the inference phase, DOGR first generates identifiers for the given query, then the documents are ranked according to the relevance scores derived from generation probabilities of identifiers and the semantic scores.