Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Correlating Factors for Domain Adaptation Performance

Goksenin Yuksel, Jaap Kamps

TL;DR

This work addresses the robustness gap of dense retrievers under domain shifts by examining correlating factors for domain adaptation and extending prior analyses of zero-shot performance. It primarily evaluates GPL as a domain-adaptation method and conducts a case study with InPars, across BEIR and LoTTE datasets, while introducing generated-query attributes such as vocabulary overlap and query-type distribution entropy. The key contributions are: (i) demonstrating robust cross-domain gains with GPL, (ii) identifying generated-query-type entropy and test-domain overlap as strong predictors of improvement, and (iii) revealing framework-specific differences in how synthetic data align with target domains. The findings provide actionable guidance for crafting domain-tailored synthetic data to enhance the robustness of dense retrievers in diverse, real-world domains.

Abstract

Dense retrievers have demonstrated significant potential for neural information retrieval; however, they lack robustness to domain shifts, limiting their efficacy in zero-shot settings across diverse domains. In this paper, we set out to analyze the possible factors that lead to successful domain adaptation of dense retrievers. We include domain similarity proxies between generated queries to test and source domains. Furthermore, we conduct a case study comparing two powerful domain adaptation techniques. We find that generated query type distribution is an important factor, and generating queries that share a similar domain to the test documents improves the performance of domain adaptation methods. This study further emphasizes the importance of domain-tailored generated queries.

On Correlating Factors for Domain Adaptation Performance

TL;DR

This work addresses the robustness gap of dense retrievers under domain shifts by examining correlating factors for domain adaptation and extending prior analyses of zero-shot performance. It primarily evaluates GPL as a domain-adaptation method and conducts a case study with InPars, across BEIR and LoTTE datasets, while introducing generated-query attributes such as vocabulary overlap and query-type distribution entropy. The key contributions are: (i) demonstrating robust cross-domain gains with GPL, (ii) identifying generated-query-type entropy and test-domain overlap as strong predictors of improvement, and (iii) revealing framework-specific differences in how synthetic data align with target domains. The findings provide actionable guidance for crafting domain-tailored synthetic data to enhance the robustness of dense retrievers in diverse, real-world domains.

Abstract

Dense retrievers have demonstrated significant potential for neural information retrieval; however, they lack robustness to domain shifts, limiting their efficacy in zero-shot settings across diverse domains. In this paper, we set out to analyze the possible factors that lead to successful domain adaptation of dense retrievers. We include domain similarity proxies between generated queries to test and source domains. Furthermore, we conduct a case study comparing two powerful domain adaptation techniques. We find that generated query type distribution is an important factor, and generating queries that share a similar domain to the test documents improves the performance of domain adaptation methods. This study further emphasizes the importance of domain-tailored generated queries.
Paper Structure (11 sections, 3 figures, 1 table)

This paper contains 11 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Research
  3. Method
  4. Results
  5. GPL Domain Adaptation
  6. Performance Factors
  7. Case Study
  8. Discussion and Conclusions
  9. Limitations
  10. Query Types
  11. Data, code, and trained models

Figures (3)

  • Figure 1: Comparison of factors affecting domain adaptation performance measured in NDCG@10 improvement.
  • Figure 2: Generated query type distribution for BEIR dataset
  • Figure 3: Factors for test to generated domain overlap in BEIR datasets. Two generated domain are compared, GPL and InPars.