Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Streamlining Conformal Information Retrieval via Score Refinement

Yotam Intrator, Ori Kelner, Regev Cohen, Roman Goldenberg, Ehud Rivlin, Daniel Freedman

TL;DR

This work introduces a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees.

Abstract

Information retrieval (IR) methods, like retrieval augmented generation, are fundamental to modern applications but often lack statistical guarantees. Conformal prediction addresses this by retrieving sets guaranteed to include relevant information, yet existing approaches produce large-sized sets, incurring high computational costs and slow response times. In this work, we introduce a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees. Experiments on various BEIR benchmarks validate the effectiveness of our approach in producing compact sets containing relevant information.

Streamlining Conformal Information Retrieval via Score Refinement

TL;DR

This work introduces a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees.

Abstract

Information retrieval (IR) methods, like retrieval augmented generation, are fundamental to modern applications but often lack statistical guarantees. Conformal prediction addresses this by retrieving sets guaranteed to include relevant information, yet existing approaches produce large-sized sets, incurring high computational costs and slow response times. In this work, we introduce a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees. Experiments on various BEIR benchmarks validate the effectiveness of our approach in producing compact sets containing relevant information.
Paper Structure (11 sections, 6 equations, 3 figures, 4 tables)

This paper contains 11 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Information Retrieval: Overview
  4. Conformal Prediction for Retrieval
  5. Method
  6. Experiments
  7. Setup
  8. Results
  9. Conclusion
  10. Limitations
  11. Additional Experiments

Figures (3)

  • Figure 1: Retrieval Pipeline. The query is first embedded using a semantic embedder, and then the top $N$ candidates are retrieved from a vector store. Crucially, their corresponding scores then undergo a refinement transformation before being passed through a conformal prediction method that outputs an adaptive set of documents.
  • Figure 2: Impact of $\lambda$ value on average group size using BGE-large-1.5 on SCIFACT with $\alpha$ = 0.05.
  • Figure 3: Performance comparison using BGE-large-1.5 on FEVER dataset across various values of $\alpha$.