Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot Dense Retrieval with Embeddings from Relevance Feedback

Nour Jedidi, Yung-Sung Chuang, Leslie Shing, James Glass

TL;DR

Inspired by relevance feedback, ReDE-RF proposes to re-frame hypothetical document generation as a relevance estimation task, using an LLM to select which documents should be used for nearest neighbor search while also making significant improvements in latency per-query.

Abstract

Building effective dense retrieval systems remains difficult when relevance supervision is not available. Recent work has looked to overcome this challenge by using a Large Language Model (LLM) to generate hypothetical documents that can be used to find the closest real document. However, this approach relies solely on the LLM to have domain-specific knowledge relevant to the query, which may not be practical. Furthermore, generating hypothetical documents can be inefficient as it requires the LLM to generate a large number of tokens for each query. To address these challenges, we introduce Real Document Embeddings from Relevance Feedback (ReDE-RF). Inspired by relevance feedback, ReDE-RF proposes to re-frame hypothetical document generation as a relevance estimation task, using an LLM to select which documents should be used for nearest neighbor search. Through this re-framing, the LLM no longer needs domain-specific knowledge but only needs to judge what is relevant. Additionally, relevance estimation only requires the LLM to output a single token, thereby improving search latency. Our experiments show that ReDE-RF consistently surpasses state-of-the-art zero-shot dense retrieval methods across a wide range of low-resource retrieval datasets while also making significant improvements in latency per-query.

Zero-Shot Dense Retrieval with Embeddings from Relevance Feedback

TL;DR

Inspired by relevance feedback, ReDE-RF proposes to re-frame hypothetical document generation as a relevance estimation task, using an LLM to select which documents should be used for nearest neighbor search while also making significant improvements in latency per-query.

Abstract

Building effective dense retrieval systems remains difficult when relevance supervision is not available. Recent work has looked to overcome this challenge by using a Large Language Model (LLM) to generate hypothetical documents that can be used to find the closest real document. However, this approach relies solely on the LLM to have domain-specific knowledge relevant to the query, which may not be practical. Furthermore, generating hypothetical documents can be inefficient as it requires the LLM to generate a large number of tokens for each query. To address these challenges, we introduce Real Document Embeddings from Relevance Feedback (ReDE-RF). Inspired by relevance feedback, ReDE-RF proposes to re-frame hypothetical document generation as a relevance estimation task, using an LLM to select which documents should be used for nearest neighbor search. Through this re-framing, the LLM no longer needs domain-specific knowledge but only needs to judge what is relevant. Additionally, relevance estimation only requires the LLM to output a single token, thereby improving search latency. Our experiments show that ReDE-RF consistently surpasses state-of-the-art zero-shot dense retrieval methods across a wide range of low-resource retrieval datasets while also making significant improvements in latency per-query.

Paper Structure

This paper contains 27 sections, 6 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries: HyDE
  4. ReDE-RF
  5. Relevance Feedback with LLMs
  6. Updating the Query Representation
  7. Experiments
  8. Setup
  9. Results on Benchmarks
  10. Comparing Latencies
  11. Ablation Study on ReDE-RF
  12. Can we Distill ReDE-RF?
  13. ReDE-RF vs. Pointwise Reranking
  14. Related Work
  15. Conclusion
  16. ...and 12 more sections

Figures (4)

  • Figure 1: An illustration of the ReDE-RF approach.
  • Figure 2: Prompt for relevance feedback, which is a modified version of the prompt used in upadhyay2024llms. {} denotes the placeholder for the corresponding text.
  • Figure 3: Latency per query for HyDE$_\text{PRF}$, HyDE and ReDE-RF. Speedup is relative to slowest method (HyDE$_\text{PRF}$).
  • Figure 4: Comparison of DistillReDE to Contriever and ReDE-RF (Default: HyDE$_\text{PRF}$).