Towards Better Instruction Following Retrieval Models

TL;DR

This work tackles the challenge of instruction-following in information retrieval by introducing InF-IR, a large-scale corpus that converts standard query-passage pairs into expressive <instruction, query, passage> triplets and supplies hard negatives validated by an advanced model. Building on InF-IR, the authors propose InF-Embed, an instruction-aware embedding model trained with contrastive learning and instruction-query attention to align embeddings with user intents. Across five instruction-based benchmarks, InF-Embed significantly improves retrieval performance for embedding-based and auto-regressive backbones, illustrating the value of high-quality instruction-aware training data and robust contrastive objectives. The work also provides extensive ablations and analysis to guide future development of scalable, instruction-following retrieval systems.

Abstract

Modern information retrieval (IR) models, trained exclusively on standard <query, passage> pairs, struggle to effectively interpret and follow explicit user instructions. We introduce InF-IR, a large-scale, high-quality training corpus tailored for enhancing retrieval models in Instruction-Following IR. InF-IR expands traditional training pairs into over 38,000 expressive <instruction, query, passage> triplets as positive samples. In particular, for each positive triplet, we generate two additional hard negative examples by poisoning both instructions and queries, then rigorously validated by an advanced reasoning model (o3-mini) to ensure semantic plausibility while maintaining instructional incorrectness. Unlike existing corpora that primarily support computationally intensive reranking tasks for decoder-only language models, the highly contrastive positive-negative triplets in InF-IR further enable efficient representation learning for smaller encoder-only models, facilitating direct embedding-based retrieval. Using this corpus, we train InF-Embed, an instruction-aware Embedding model optimized through contrastive learning and instruction-query attention mechanisms to align retrieval outcomes precisely with user intents. Extensive experiments across five instruction-based retrieval benchmarks demonstrate that InF-Embed significantly surpasses competitive baselines by 8.1% in p-MRR, measuring the instruction-following capabilities.

Figures (6)

• Figure 1: Example of original information retrieval compared to instruction-following retrieval.
• Figure 2: Hard negative samples in InF-IR generated by poisoning both instructions and queries.
• Figure 3: Visualization and diversity analysis of synthetic training samples from InF-IR.
• Figure 4: Cohen's kappa from 100 random samples.
• Figure 5: Comparative analysis of instruction-following capabilities on the Follow-IR benchmark across model architectures of varying scales. Models are grouped by parameter count and ranked by p-MRR scores within each category. Standard retrieval metrics (score$*$) are normalized by a factor of 10 to facilitate visual comparison with $p$-MRR values.