Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models

Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, Jack Hessel

TL;DR

<3-5 sentence high-level summary> Promptriever introduces a promptable dense retriever trained on a large, instruction-focused MS MARCO augmentation, enabling per-query instruction control over relevance. It demonstrates strong instruction-following capabilities, competitive standard retrieval, and reliable gains when prompted at test time, bridging LM prompting methods with information retrieval. The work shows that instruction negatives and joint MS MARCO data significantly contribute to learning to condition relevance on natural language instructions, and that prompts can meaningfully improve out-of-domain retrieval. Overall, it paves the way for controllable, instruction-driven retrieval and cross-pollination between LM prompting and IR research.

Abstract

Instruction-tuned language models (LM) are able to respond to imperative commands, providing a more natural user interface compared to their base counterparts. In this work, we present Promptriever, the first retrieval model able to be prompted like an LM. To train Promptriever, we curate and release a new instance-level instruction training set from MS MARCO, spanning nearly 500k instances. Promptriever not only achieves strong performance on standard retrieval tasks, but also follows instructions. We observe: (1) large gains (reaching SoTA) on following detailed relevance instructions (+14.3 p-MRR / +3.1 nDCG on FollowIR), (2) significantly increased robustness to lexical choices/phrasing in the query+instruction (+12.9 Robustness@10 on InstructIR), and (3) the ability to perform hyperparameter search via prompting to reliably improve retrieval performance (+1.4 average increase on BEIR). Promptriever demonstrates that retrieval models can be controlled with prompts on a per-query basis, setting the stage for future work aligning LM prompting techniques with information retrieval.

Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models

TL;DR

<3-5 sentence high-level summary> Promptriever introduces a promptable dense retriever trained on a large, instruction-focused MS MARCO augmentation, enabling per-query instruction control over relevance. It demonstrates strong instruction-following capabilities, competitive standard retrieval, and reliable gains when prompted at test time, bridging LM prompting methods with information retrieval. The work shows that instruction negatives and joint MS MARCO data significantly contribute to learning to condition relevance on natural language instructions, and that prompts can meaningfully improve out-of-domain retrieval. Overall, it paves the way for controllable, instruction-driven retrieval and cross-pollination between LM prompting and IR research.

Abstract

Instruction-tuned language models (LM) are able to respond to imperative commands, providing a more natural user interface compared to their base counterparts. In this work, we present Promptriever, the first retrieval model able to be prompted like an LM. To train Promptriever, we curate and release a new instance-level instruction training set from MS MARCO, spanning nearly 500k instances. Promptriever not only achieves strong performance on standard retrieval tasks, but also follows instructions. We observe: (1) large gains (reaching SoTA) on following detailed relevance instructions (+14.3 p-MRR / +3.1 nDCG on FollowIR), (2) significantly increased robustness to lexical choices/phrasing in the query+instruction (+12.9 Robustness@10 on InstructIR), and (3) the ability to perform hyperparameter search via prompting to reliably improve retrieval performance (+1.4 average increase on BEIR). Promptriever demonstrates that retrieval models can be controlled with prompts on a per-query basis, setting the stage for future work aligning LM prompting techniques with information retrieval.
Paper Structure (34 sections, 5 figures, 13 tables)

This paper contains 34 sections, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Promptriever Data Generation
  3. Instruction Generation
  4. Instruction diversity
  5. Maintaining original MS MARCO positive relevance
  6. Instruction Negative Mining
  7. Filtering validation
  8. Experimental Settings
  9. Training
  10. Evaluation Datasets
  11. Results
  12. Instruction Following
  13. Standard Retrieval
  14. Retrieval with Prompts
  15. Analysis
  16. ...and 19 more sections

Figures (5)

  • Figure 1: An illustration of the capabilities of retrieval models. Standard retrieval models find semantic similarity to the input query, typically matching using query keywords and phrases. Current instructable retrievers prepend a dataset prefix that generically describes the task and is also used in training. We propose promptable retrievers which can handle complex instructions including detailed relevance definitions and zero-shot prompting techniques that act as a form of zero-shot hyperparameter optimization, similar to prompting LMs.
  • Figure 2: The data generation process to generate instruction-based retrieval data. We take the initial query and relevant passage and prompt an LM to generate an instruction that would match that query. Note that the instruction adds extra qualifications to the definition of relevance. We then ask an LM to generate an example relevant and non-relevant passage for that query and instruction. We see that the generated positive passage fulfills the extra requirement (in pink) but the generated instruction-negative does not. We generate multiple types of instructions (both in length and style) for training set diversity.
  • Figure 3: Prompt for Instruction Generation
  • Figure 4: Prompt for Instruction Negatives
  • Figure 5: System Prompt for All Prompts