Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Domain Adaptation for Dense Retrieval and Conversational Dense Retrieval through Self-Supervision by Meticulous Pseudo-Relevance Labeling

Minghan Li, Eric Gaussier

TL;DR

This paper proposes to combine the query-generation approach with a self-supervision approach in which pseudo-relevance labels are automatically generated on the target domain, enabling a model’s domain adaptation with real queries and documents from the target dataset.

Abstract

Recent studies have demonstrated that the ability of dense retrieval models to generalize to target domains with different distributions is limited, which contrasts with the results obtained with interaction-based models. Prior attempts to mitigate this challenge involved leveraging adversarial learning and query generation approaches, but both approaches nevertheless resulted in limited improvements. In this paper, we propose to combine the query-generation approach with a self-supervision approach in which pseudo-relevance labels are automatically generated on the target domain. To accomplish this, a T5-3B model is utilized for pseudo-positive labeling, and meticulous hard negatives are chosen. We also apply this strategy on conversational dense retrieval model for conversational search. A similar pseudo-labeling approach is used, but with the addition of a query-rewriting module to rewrite conversational queries for subsequent labeling. This proposed approach enables a model's domain adaptation with real queries and documents from the target dataset. Experiments on standard dense retrieval and conversational dense retrieval models both demonstrate improvements on baseline models when they are fine-tuned on the pseudo-relevance labeled data.

Domain Adaptation for Dense Retrieval and Conversational Dense Retrieval through Self-Supervision by Meticulous Pseudo-Relevance Labeling

TL;DR

This paper proposes to combine the query-generation approach with a self-supervision approach in which pseudo-relevance labels are automatically generated on the target domain, enabling a model’s domain adaptation with real queries and documents from the target dataset.

Abstract

Recent studies have demonstrated that the ability of dense retrieval models to generalize to target domains with different distributions is limited, which contrasts with the results obtained with interaction-based models. Prior attempts to mitigate this challenge involved leveraging adversarial learning and query generation approaches, but both approaches nevertheless resulted in limited improvements. In this paper, we propose to combine the query-generation approach with a self-supervision approach in which pseudo-relevance labels are automatically generated on the target domain. To accomplish this, a T5-3B model is utilized for pseudo-positive labeling, and meticulous hard negatives are chosen. We also apply this strategy on conversational dense retrieval model for conversational search. A similar pseudo-labeling approach is used, but with the addition of a query-rewriting module to rewrite conversational queries for subsequent labeling. This proposed approach enables a model's domain adaptation with real queries and documents from the target dataset. Experiments on standard dense retrieval and conversational dense retrieval models both demonstrate improvements on baseline models when they are fine-tuned on the pseudo-relevance labeled data.
Paper Structure (32 sections, 4 equations, 5 figures, 4 tables)

This paper contains 32 sections, 4 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pseudo-Quesries and Pseudo-Labeling for DR or IR
  4. Conversational Dense Retrieval
  5. Background
  6. Dense retrieval
  7. BM25
  8. T53B
  9. Conversational Dense Retrieval
  10. Pseudo-Relevance Labeling for Dense Retrieval
  11. Pseudo-Positive Sampling
  12. Pseudo-Negative Sampling
  13. Global and BM25 Hard Negative Sampling
  14. Meticulous Hard Negative Sampling
  15. Improving GPL: Combining Pseudo-Relevance Labels and Pseudo-Queries
  16. ...and 17 more sections

Figures (5)

  • Figure 1: An example of conversational search (with our model deployed, top 1 as answer).
  • Figure 2: The overall pipeline with BM25 hard negative sampling for pseudo-relevance labeling.
  • Figure 3: The overall pipeline of generating self-supervised data with meticulous pseudo-relevance labeling using SimANS hard negative sampling.
  • Figure 4: CDR architecture with training.
  • Figure 5: Overall pipeline of generating pseudo-data for conversational dense retrieval.