Can Instructed Retrieval Models Really Support Exploration?

TL;DR

The paper investigates whether instructed retrieval models can support seed-guided exploration in scientific corpora by evaluating ranking relevance and instruction following on the CSFCube dataset using various models, including PRP-enabled LLMs and instructed retrievers. It finds that while instructed retrievers can improve $NDCG@20$ over instruction-agnostic baselines, their instruction-following behavior often does not align with ranking gains and can be counter-intuitive or insensitive, raising questions about their suitability for long exploratory sessions. The study provides a robust, expert-annotated testbed and highlights a decoupling between ranking quality and user-facing instruction adherence, suggesting direction for future human-centered improvements. Overall, the work offers empirical evidence and a framework for evaluating instruction following in exploratory retrieval, with implications for building more responsive exploratory search systems.

Abstract

Exploratory searches are characterized by under-specified goals and evolving query intents. In such scenarios, retrieval models that can capture user-specified nuances in query intent and adapt results accordingly are desirable -- instruction-following retrieval models promise such a capability. In this work, we evaluate instructed retrievers for the prevalent yet under-explored application of aspect-conditional seed-guided exploration using an expert-annotated test collection. We evaluate both recent LLMs fine-tuned for instructed retrieval and general-purpose LLMs prompted for ranking with the highly performant Pairwise Ranking Prompting. We find that the best instructed retrievers improve on ranking relevance compared to instruction-agnostic approaches. However, we also find that instruction following performance, crucial to the user experience of interacting with models, does not mirror ranking relevance improvements and displays insensitivity or counter-intuitive behavior to instructions. Our results indicate that while users may benefit from using current instructed retrievers over instruction-agnostic models, they may not benefit from using them for long-running exploratory sessions requiring greater sensitivity to instructions.

Figures (1)

• Figure 1: CSFCube contains the same query document annotated by experts w.r.t candidate documents for relevance with different instructions. This enables evaluation of both ranking relevance and instruction following for instructed retrievers.