Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PaRaDe: Passage Ranking using Demonstrations with Large Language Models

Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, Kai Hui

TL;DR

PaRaDe tackles the sensitivity of LLM-driven passage re-ranking to the demonstrations included in prompts. It introduces difficulty-based selection (DBS) to automatically pick challenging demonstrations, showing DBS significantly improves ranking over zero-shot and randomly selected demonstrations on TREC/BEIR benchmarks, and extends the approach to joint demonstration selection (CDQL) and to question generation. The work demonstrates that careful demonstration selection can yield substantial gains and provides a practical, automatic method to enhance prompt-based ranking without task-specific fine-tuning. It also reveals limitations of similarity-based selection and highlights opportunities for future work combining DBS with other selection signals and applying to broader ranking tasks.

Abstract

Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance. In this work, we improve LLM-based re-ranking by algorithmically selecting few-shot demonstrations to include in the prompt. Our analysis investigates the conditions where demonstrations are most helpful, and shows that adding even one demonstration is significantly beneficial. We propose a novel demonstration selection strategy based on difficulty rather than the commonly used semantic similarity. Furthermore, we find that demonstrations helpful for ranking are also effective at question generation. We hope our work will spur more principled research into question generation and passage ranking.

PaRaDe: Passage Ranking using Demonstrations with Large Language Models

TL;DR

PaRaDe tackles the sensitivity of LLM-driven passage re-ranking to the demonstrations included in prompts. It introduces difficulty-based selection (DBS) to automatically pick challenging demonstrations, showing DBS significantly improves ranking over zero-shot and randomly selected demonstrations on TREC/BEIR benchmarks, and extends the approach to joint demonstration selection (CDQL) and to question generation. The work demonstrates that careful demonstration selection can yield substantial gains and provides a practical, automatic method to enhance prompt-based ranking without task-specific fine-tuning. It also reveals limitations of similarity-based selection and highlights opportunities for future work combining DBS with other selection signals and applying to broader ranking tasks.

Abstract

Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance. In this work, we improve LLM-based re-ranking by algorithmically selecting few-shot demonstrations to include in the prompt. Our analysis investigates the conditions where demonstrations are most helpful, and shows that adding even one demonstration is significantly beneficial. We propose a novel demonstration selection strategy based on difficulty rather than the commonly used semantic similarity. Furthermore, we find that demonstrations helpful for ranking are also effective at question generation. We hope our work will spur more principled research into question generation and passage ranking.
Paper Structure (25 sections, 4 equations, 4 figures, 6 tables)

This paper contains 25 sections, 4 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. LLM Re-ranking by Query Likelihood
  3. Background.
  4. Experiments on TREC and BEIR
  5. Setup.
  6. Demonstrations for Ranking
  7. The Impact of Demonstrations
  8. Demonstrations influence ranking.
  9. Increasing demonstrations does not necessarily help.
  10. Similarity-based selection is limited.
  11. Difficulty-based Selection (DBS)
  12. Difficult Demonstrations are Beneficial
  13. Main Results and Discussion
  14. DBS for TREC and BEIR
  15. Demonstration filtering.
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Statistics for nDCG@10 on TREC 2020, aggregated using the same query with 100 different one-shot demonstrations. Flan-T5-XXL is used for re-ranking. Zero-shot results included for reference.
  • Figure 2: Statistics for nDCG@10 on TREC 2020, aggregated using 100 different $k$-shot demonstrations with Flan-T5-XL and XXL models. Number of demonstrations ($k$) shown after the dash.
  • Figure 3: The nDCG@10 of DQL-based bins measured on TREC 2020. The x-axis increases in difficulty of demonstration from left-to-right.
  • Figure 4: For each demonstration, we compute the semantic similarity between ground-truth and the synthetic queries. We first measure the max similarity by demonstration and query. Then we average this across all queries, giving a single scalar per demonstration. The dashed line shows the "average max similarity" for the demonstration chosen using DBS.