Sherlock Your Queries: Learning to Ask the Right Questions for Dialogue-Based Retrieval

TL;DR

This work tackles ambiguity in user queries by learning a dynamic sequence of yes/no questions to drive dialogue-based retrieval. SherlockLLM couples an LLM-based Questioner with a domain-specific Retriever and trains the policy via Group Relative Policy Optimization, enabling efficient information gathering without large annotated dialogue data. Across structured tasks (Guess Number, Guess Who) and unstructured image retrieval (CelebA), the method nearly reaches binary-search optimality in structured settings and significantly outperforms strong baselines in unstructured retrieval, with ablations underscoring the importance of information-gain–driven rewards and retrieval feedback. The contributions offer a scalable, modular approach to interactive IR with potential broad applicability across modalities and domains.

Abstract

User queries in information retrieval are often ambiguous, making it challenging for systems to identify a user's target from a single query. While recent dialogue-based interactive retrieval systems can clarify user intent, they are inefficient as they often lack an explicit strategy to ask the most informative questions. To address this limitation, we propose SherlockLLM, a dialogue-driven retrieval framework that learns an optimal questioning strategy via Reinforcement Learning (RL) and avoids the need for large-scale annotated dialogue data. In our framework, an agent is trained to generate a sequence of binary questions to efficiently narrow down the search space. To validate our approach, we introduce a benchmark with both structured and unstructured tasks. Experimental results show that SherlockLLM is a robust and efficient solution. On the structured tasks, its performance matches strong baselines and approaches the theoretical optimal defined by binary search. On the challenging unstructured task, our agent significantly outperforms these baselines, showcasing its ability to learn a highly effective information-seeking dialogue policy.

Figures (10)

• Figure 1: Our dialogue-based information retrieval agent. Without precise target queries, the agent identifies the desired target through a series of yes/no questions. At each turn, the agent generates the most informative question for the user, minimizing the number of interactions needed to determine the target.
• Figure 2: The overview of the pipeline. An LLM fine-tuned using reinforcement learning acts as a Questioner to ask questions, while the User answers questions as feedback.
• Figure 3: Examples from the Guess Number and Guess Who tasks. (a) illustrates the agent's strategy in Guess Number, while (b) shows its approach in Guess Who.
• Figure 4: Examples from the image retrieval task. For each turn, we show the target's rank, EIG of the chosen question, and its EIG Rank among all possible questions for the top-10 candidates.
• Figure 5: Examples of the Guess Number performed by baseline models and SherlockLLM.