Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning

Li Siyan, Darshan Deshpande, Anand Kannappan, Rebecca Qian

TL;DR

DETOUR introduces a dual-agent, interactive benchmark for tip-of-the-tongue retrieval with 1,011 multimodal prompts across eight domains. It grounds a Memory Agent in memory files while a Primary Agent performs reasoning and web search, enabling multi-turn clarification evaluation. The study shows state-of-the-art models struggle on multimodal ToT tasks, with all-modal accuracy around 36% and substantial variability across models and prompts, highlighting the need for better memory grounding and clarification strategies. DETOUR provides a foundation for advancing interactive, memory-grounded agentic search and clarifying question capabilities in real-world conversations.

Abstract

When recalling information in conversation, people often arrive at the recollection after multiple turns. However, existing benchmarks for evaluating agent capabilities in such tip-of-the-tongue search processes are restricted to single-turn settings. To more realistically simulate tip-of-the-tongue search, we introduce Dual-agent based Evaluation Through Obscure Under-specified Retrieval (DETOUR), a dual-agent evaluation benchmark containing 1,011 prompts. The benchmark design involves a Primary Agent, which is the subject of evaluation, tasked with identifying the recollected entity through querying a Memory Agent that is held consistent across evaluations. Our results indicate that current state-of-the-art models still struggle with our benchmark, only achieving 36% accuracy when evaluated on all modalities (text, image, audio, and video), highlighting the importance of enhancing capabilities in underspecified scenarios.

DETOUR: An Interactive Benchmark for Dual-Agent Search and Reasoning

TL;DR

DETOUR introduces a dual-agent, interactive benchmark for tip-of-the-tongue retrieval with 1,011 multimodal prompts across eight domains. It grounds a Memory Agent in memory files while a Primary Agent performs reasoning and web search, enabling multi-turn clarification evaluation. The study shows state-of-the-art models struggle on multimodal ToT tasks, with all-modal accuracy around 36% and substantial variability across models and prompts, highlighting the need for better memory grounding and clarification strategies. DETOUR provides a foundation for advancing interactive, memory-grounded agentic search and clarifying question capabilities in real-world conversations.

Abstract

When recalling information in conversation, people often arrive at the recollection after multiple turns. However, existing benchmarks for evaluating agent capabilities in such tip-of-the-tongue search processes are restricted to single-turn settings. To more realistically simulate tip-of-the-tongue search, we introduce Dual-agent based Evaluation Through Obscure Under-specified Retrieval (DETOUR), a dual-agent evaluation benchmark containing 1,011 prompts. The benchmark design involves a Primary Agent, which is the subject of evaluation, tasked with identifying the recollected entity through querying a Memory Agent that is held consistent across evaluations. Our results indicate that current state-of-the-art models still struggle with our benchmark, only achieving 36% accuracy when evaluated on all modalities (text, image, audio, and video), highlighting the importance of enhancing capabilities in underspecified scenarios.
Paper Structure (32 sections, 1 equation, 5 figures, 4 tables)

This paper contains 32 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agentic Search Evaluation
  4. Searching Under Uncertainty
  5. Interactive Benchmarking of Agents
  6. DETOUR
  7. Data Collection
  8. Benchmark Composition
  9. Evaluation Harness
  10. Evaluating Primary Agent Correctness
  11. Memory Agent Construction
  12. Experiments and Discussion
  13. Evaluation on Full Harness
  14. Error Analysis
  15. Ablation: Memory Agent Query Tool
  16. ...and 17 more sections

Figures (5)

  • Figure 1: An example instance from DETOUR. Here, the input file shows the Universal Studios Globe, signaling a Universal release. The Memory File includes an astronaut, hinting that the film is set in space. Tying the amnesia action-thriller clue to The Bourne Identity leads to Matt Damon, and then to The Martian, where his character fights to survive on Mars.
  • Figure 2: The overview of our dual-agent tip-of-the-tongue known-item search and retrieval process.
  • Figure 3: Distributions of input modality (left), memory file modality (middle), and question domains (right).
  • Figure 4: Query-type composition conditioned on model correctness. For each model, we show the proportion of questions that are fully correct ("Correct Question") versus partially correct ("Partially Correct Question"), separately for cases where the model's final answer is correct (left) and incorrect (right). Bars are normalized within each model and correctness condition, so each row sums to 100%.
  • Figure 5: The results from our in-context example ablation experiment.