RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark

TL;DR

RECOR addresses the need for simultaneously handling multi-turn dialogue and reasoning-based retrieval by introducing a Decomposition-and-Verification framework that produces fact-grounded, multi-turn dialogues across eleven domains. The benchmark combines BRIGHT and StackExchange data to form 707 conversations and 2,971 turns, with explicit turn-level reasoning and verification against sources. Empirical results show that incorporating conversation history with explicit reasoning doubles retrieval performance ($nDCG@10$), and reasoning-specialized retrievers outperform dense encoders, though implicit, unstated reasoning remains challenging. The work provides a valuable resource for developing reasoning-guided CIR systems and highlights opportunities to improve implicit reasoning in retrieval and generation tasks.

Abstract

Existing benchmarks treat multi-turn conversation and reasoning-intensive retrieval separately, yet real-world information seeking requires both. To bridge this gap, we present a benchmark for reasoning-based conversational information retrieval comprising 707 conversations (2,971 turns) across eleven domains. To ensure quality, our Decomposition-and-Verification framework transforms complex queries into fact-grounded multi-turn dialogues through multi-level validation, where atomic facts are verified against sources and explicit retrieval reasoning is generated for each turn. Comprehensive evaluation reveals that combining conversation history with reasoning doubles retrieval performance (Baseline .236 $\rightarrow$ History+Reasoning .479 nDCG@10), while reasoning-specialized models substantially outperform dense encoders. Despite these gains, further analysis highlights that implicit reasoning remains challenging, particularly when logical connections are not explicitly stated in the text.

Figures (21)

• Figure 1: Overview of Decomposition-and-Verification framework for generating grounded multi-turn conversations.
• Figure 2: Retrieval performance (nDCG@10) by turn position (avg. 8 retrievers, 11 domains). Non-reasoning strategies share equal T1 performance; divergence emerges from Turn 2 as context dependence grows.
• Figure 3: Generation quality by turn position across 7 models (Retrieved mode). Large models (solid) outperform small (dotted). All show significant decline ($p < 0.01$).
• Figure 4: Prompt for validating document-answer alignment.
• Figure 5: Prompt for extracting granular aspects from gold answers.