Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ICLERB: In-Context Learning Embedding and Reranker Benchmark

Marie Al Ghossein, Emile Contal, Alexandre Robicquet

TL;DR

The paper reframes retrieval for In-Context Learning as a recommendation problem rather than a semantic search task, introducing the ICLERB benchmark to evaluate how retrieved documents improve LLM accuracy using $\mathrm{nDCG}$-based metrics derived from $DPO$ rewards. It presents RLRAIF, a reinforcement-learning-to-rank approach that uses AI feedback from LLMs to fine-tune retrievers with a budgeted number of queries, achieving state-of-the-art results with much smaller models. Empirically, small models tuned with RLRAIF can outperform larger, conventionally trained retrieval systems on ICL tasks, highlighting the misalignment between traditional retrieval benchmarks (e.g., MTEB) and ICL objectives. The work emphasizes the need for specialized benchmarks and training strategies tailored to ICL, and outlines plans to expand ICLERB to more datasets, LLMs, and retrieval architectures for broader applicability.

Abstract

In-Context Learning (ICL) enables Large Language Models (LLMs) to perform new tasks by conditioning on prompts with relevant information. Retrieval-Augmented Generation (RAG) enhances ICL by incorporating retrieved documents into the LLM's context at query time. However, traditional retrieval methods focus on semantic relevance, treating retrieval as a search problem. In this paper, we propose reframing retrieval for ICL as a recommendation problem, aiming to select documents that maximize utility in ICL tasks. We introduce the In-Context Learning Embedding and Reranker Benchmark (ICLERB), a novel evaluation framework that compares retrievers based on their ability to enhance LLM accuracy in ICL settings. Additionally, we propose a novel Reinforcement Learning-to-Rank from AI Feedback (RLRAIF) algorithm, designed to fine-tune retrieval models using minimal feedback from the LLM. Our experimental results reveal notable differences between ICLERB and existing benchmarks, and demonstrate that small models fine-tuned with our RLRAIF algorithm outperform large state-of-the-art retrieval models. These findings highlight the limitations of existing evaluation methods and the need for specialized benchmarks and training strategies adapted to ICL.

ICLERB: In-Context Learning Embedding and Reranker Benchmark

TL;DR

The paper reframes retrieval for In-Context Learning as a recommendation problem rather than a semantic search task, introducing the ICLERB benchmark to evaluate how retrieved documents improve LLM accuracy using -based metrics derived from rewards. It presents RLRAIF, a reinforcement-learning-to-rank approach that uses AI feedback from LLMs to fine-tune retrievers with a budgeted number of queries, achieving state-of-the-art results with much smaller models. Empirically, small models tuned with RLRAIF can outperform larger, conventionally trained retrieval systems on ICL tasks, highlighting the misalignment between traditional retrieval benchmarks (e.g., MTEB) and ICL objectives. The work emphasizes the need for specialized benchmarks and training strategies tailored to ICL, and outlines plans to expand ICLERB to more datasets, LLMs, and retrieval architectures for broader applicability.

Abstract

In-Context Learning (ICL) enables Large Language Models (LLMs) to perform new tasks by conditioning on prompts with relevant information. Retrieval-Augmented Generation (RAG) enhances ICL by incorporating retrieved documents into the LLM's context at query time. However, traditional retrieval methods focus on semantic relevance, treating retrieval as a search problem. In this paper, we propose reframing retrieval for ICL as a recommendation problem, aiming to select documents that maximize utility in ICL tasks. We introduce the In-Context Learning Embedding and Reranker Benchmark (ICLERB), a novel evaluation framework that compares retrievers based on their ability to enhance LLM accuracy in ICL settings. Additionally, we propose a novel Reinforcement Learning-to-Rank from AI Feedback (RLRAIF) algorithm, designed to fine-tune retrieval models using minimal feedback from the LLM. Our experimental results reveal notable differences between ICLERB and existing benchmarks, and demonstrate that small models fine-tuned with our RLRAIF algorithm outperform large state-of-the-art retrieval models. These findings highlight the limitations of existing evaluation methods and the need for specialized benchmarks and training strategies adapted to ICL.

Paper Structure

This paper contains 51 sections, 12 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. In-Context Learning and Retrieval-Augmented Generation
  4. Limitations of Unsupervised Retrieval Methods Based on Semantic Similarity
  5. Text Embeddings and Cosine Similarity
  6. Pre-trained Embeddings for Semantic Similarity
  7. Pre-training with Weak Supervision on Semantic Similarity
  8. Rerankers Trained on Binary Query Relevance
  9. Limitations of Semantic Similarity for RAG ICL
  10. Retrieval Methods Supervised for Few-Shot ICL
  11. Retrieval Models Supervised on Downstream Tasks
  12. Challenges in Evaluating Retrievers Separately
  13. Fine-Tuning and DPO
  14. Direct Preference Optimization (DPO)
  15. Computational Considerations
  16. ...and 36 more sections

Figures (2)

  • Figure 1: Retrieval performance for the task of ICL, as measured by nDCG@10 in the In-Context Learning Embedding and Reranker Benchmark (ICLERB), vs. model size in millions of parameters. cm-rerank-mxbai-rlaif-v0.1 (ours) is fine-tuned by RLRAIF and outperforms other approaches while being considerably smaller than most models.
  • Figure 2: ICLERB rank reached with different DPO budgets, for RLRAIF and an approach randomly querying (query, document) pairs