Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classification

Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani, Mohammed M. Abdelgwad, Adam Jatowt

TL;DR

DynRank tackles static-prompt limitations in open-domain QA by using dynamic, fine-grained question classification to generate contextually tailored prompts for passage re-ranking. The approach integrates a retriever, a QC module, a dynamic prompt generator, and a PLM-based re-ranker in a zero-shot setting, scoring passages via the average log-likelihood of the question given the passage and prompt. Empirical results across ODQA datasets and the BEIR benchmark show consistent improvements over baselines and static prompts, including notable gains with both unsupervised and supervised retrievers and competitive performance against state-of-the-art re-ranking methods. The work demonstrates the practical impact of adaptive prompting for retrieval, with larger gains observed using more capable language models such as LLaMA v3.1 70B, underscoring the method’s scalability and cross-domain applicability.

Abstract

This paper presents DynRank, a novel framework for enhancing passage retrieval in open-domain question-answering systems through dynamic zero-shot question classification. Traditional approaches rely on static prompts and pre-defined templates, which may limit model adaptability across different questions and contexts. In contrast, DynRank introduces a dynamic prompting mechanism, leveraging a pre-trained question classification model that categorizes questions into fine-grained types. Based on these classifications, contextually relevant prompts are generated, enabling more effective passage retrieval. We integrate DynRank into existing retrieval frameworks and conduct extensive experiments on multiple QA benchmark datasets.

DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classification

TL;DR

DynRank tackles static-prompt limitations in open-domain QA by using dynamic, fine-grained question classification to generate contextually tailored prompts for passage re-ranking. The approach integrates a retriever, a QC module, a dynamic prompt generator, and a PLM-based re-ranker in a zero-shot setting, scoring passages via the average log-likelihood of the question given the passage and prompt. Empirical results across ODQA datasets and the BEIR benchmark show consistent improvements over baselines and static prompts, including notable gains with both unsupervised and supervised retrievers and competitive performance against state-of-the-art re-ranking methods. The work demonstrates the practical impact of adaptive prompting for retrieval, with larger gains observed using more capable language models such as LLaMA v3.1 70B, underscoring the method’s scalability and cross-domain applicability.

Abstract

This paper presents DynRank, a novel framework for enhancing passage retrieval in open-domain question-answering systems through dynamic zero-shot question classification. Traditional approaches rely on static prompts and pre-defined templates, which may limit model adaptability across different questions and contexts. In contrast, DynRank introduces a dynamic prompting mechanism, leveraging a pre-trained question classification model that categorizes questions into fine-grained types. Based on these classifications, contextually relevant prompts are generated, enabling more effective passage retrieval. We integrate DynRank into existing retrieval frameworks and conduct extensive experiments on multiple QA benchmark datasets.

Paper Structure

This paper contains 29 sections, 5 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Retriever
  4. Question Classification
  5. Dynamic Prompt Generation
  6. Re-ranking with Pre-trained Language Models
  7. Experiments
  8. Datasets
  9. Experimental Setup
  10. Open-Domain QA:
  11. BEIR:
  12. Implementation Details:
  13. Results
  14. Question Classification
  15. Open-Domain QA
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Top-1 Accuracy after re-ranking the top 1,000 passages retrieved by BM25 with DynRank comparing with UPR sachan2022improving on the Natural Questions TriviaQA and WebQA datasets.
  • Figure 2: The architecture of the proposed DynRank framework. First, retrieve relevant documents a retriever. The retrieved documents are then re-ranked based on a dynamically generated prompt, tailored to the question's classification into major ($L_{maj}$) and minor ($L_{min}$) types. A large language model (LLM) processes this prompt to re-rank the top-k documents, improving retrieval accuracy.
  • Figure 3: Average nDCG@10 scores for different language models on the BEIR benchmark.