Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Dynamic Dense Retrieval with Routing Strategy

Zhan Su, Fengran Mo, Jinghan Zhang, Yuchen Hui, Jia Ao Sun, Bingbing Wen, Jian-Yun Nie

TL;DR

Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2\% of the training parameters, paving the way to achieve more flexible dense retrieval in IR.

Abstract

The \textit{de facto} paradigm for applying dense retrieval (DR) to new tasks involves fine-tuning a pre-trained model for a specific task. However, this paradigm has two significant limitations: (1) It is difficult adapt the DR to a new domain if the training dataset is limited. (2) Old DR models are simply replaced by newer models that are trained from scratch when the former are no longer up to date. Especially for scenarios where the model needs to be updated frequently, this paradigm is prohibitively expensive. To address these challenges, we propose a novel dense retrieval approach, termed \textit{dynamic dense retrieval} (DDR). DDR uses \textit{prefix tuning} as a \textit{module} specialized for a specific domain. These modules can then be compositional combined with a dynamic routing strategy, enabling highly flexible domain adaptation in the retrieval part. Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2\% of the training parameters, paving the way to achieve more flexible dense retrieval in IR. We see it as a promising future direction for applying dense retrieval to various tasks.

Towards Dynamic Dense Retrieval with Routing Strategy

TL;DR

Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2\% of the training parameters, paving the way to achieve more flexible dense retrieval in IR.

Abstract

The \textit{de facto} paradigm for applying dense retrieval (DR) to new tasks involves fine-tuning a pre-trained model for a specific task. However, this paradigm has two significant limitations: (1) It is difficult adapt the DR to a new domain if the training dataset is limited. (2) Old DR models are simply replaced by newer models that are trained from scratch when the former are no longer up to date. Especially for scenarios where the model needs to be updated frequently, this paradigm is prohibitively expensive. To address these challenges, we propose a novel dense retrieval approach, termed \textit{dynamic dense retrieval} (DDR). DDR uses \textit{prefix tuning} as a \textit{module} specialized for a specific domain. These modules can then be compositional combined with a dynamic routing strategy, enabling highly flexible domain adaptation in the retrieval part. Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2\% of the training parameters, paving the way to achieve more flexible dense retrieval in IR. We see it as a promising future direction for applying dense retrieval to various tasks.
Paper Structure (16 sections, 10 equations, 1 figure, 3 tables)

This paper contains 16 sections, 10 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Prefix Tuning
  5. DDR: Dynamic Dense Retrieval with Routing Strategy
  6. Dynamic Dense Retrieval Training
  7. Routing Strategy Comparison
  8. Top-$k$ routing (DDR-topk)
  9. Routing with prior information (DDR-prior)
  10. Experimental results
  11. Datasets and Metrics
  12. Experimental Setup
  13. Baselines
  14. Experimental Results and Analysis
  15. Ablation Study
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Overview of DDR framework, built upon a dual-encoder architecture. The general prefix vector learns the general knowledge across tasks, while the domain prefix focuses on the specificity of each task domain. The frozen backbone incorporates both the general prefix and aspect prefix for query encoding, whereas the passage encoder only utilizes the general prefix.