Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PluriHop: Exhaustive, Recall-Sensitive QA over Distractor-Rich Corpora

Mykolas Sveistrys, Richard Kunert

TL;DR

This work defines pluri-hop QA as questions requiring recall sensitive, exhaustive, and exact answers across many documents in repetitive corpora. It introduces PluriHopWIND, a diagnostic multilingual dataset built from wind industry reports to stress distractors and long-range evidence aggregation, and presents PluriHopRAG, a retrieval architecture that decomposes queries to document level and uses cross encoder filtering to enable exhaustive yet efficient processing. Across comparisons with graph-based, multimodal, and naive RAG baselines, existing approaches struggle to exceed 40% statementwise F1, while PluriHopRAG achieves substantial relative gains (18–52%) depending on the base LLM, highlighting the value of exhaustive retrieval and early filtering for pluri-hop QA. Collectively, the dataset and method reveal limitations of current QA systems on repetitive, distractor-rich corpora and point to a promising direction for robust, recall-sensitive retrieval strategies in real-world document collections.

Abstract

Recent advances in large language models (LLMs) and retrieval-augmented generation (RAG) have enabled progress on question answering (QA) when relevant evidence is in one (single-hop) or multiple (multi-hop) passages. Yet many realistic questions about recurring report data - medical records, compliance filings, maintenance logs - require aggregation across all documents, with no clear stopping point for retrieval and high sensitivity to even one missed passage. We term these pluri-hop questions and formalize them by three criteria: recall sensitivity, exhaustiveness, and exactness. To study this setting, we introduce PluriHopWIND, a diagnostic multilingual dataset of 48 pluri-hop questions built from 191 real-world wind industry reports in German and English. We show that PluriHopWIND is 8-40% more repetitive than other common datasets and thus has higher density of distractor documents, better reflecting practical challenges of recurring report corpora. We test a traditional RAG pipeline as well as graph-based and multimodal variants, and find that none of the tested approaches exceed 40% in statement-wise F1 score. Motivated by this, we propose PluriHopRAG, a RAG architecture that follows a "check all documents individually, filter cheaply" approach: it (i) decomposes queries into document-level subquestions and (ii) uses a cross-encoder filter to discard irrelevant documents before costly LLM reasoning. We find that PluriHopRAG achieves relative F1 score improvements of 18-52% depending on base LLM. Despite its modest size, PluriHopWIND exposes the limitations of current QA systems on repetitive, distractor-rich corpora. PluriHopRAG's performance highlights the value of exhaustive retrieval and early filtering as a powerful alternative to top-k methods.

PluriHop: Exhaustive, Recall-Sensitive QA over Distractor-Rich Corpora

TL;DR

This work defines pluri-hop QA as questions requiring recall sensitive, exhaustive, and exact answers across many documents in repetitive corpora. It introduces PluriHopWIND, a diagnostic multilingual dataset built from wind industry reports to stress distractors and long-range evidence aggregation, and presents PluriHopRAG, a retrieval architecture that decomposes queries to document level and uses cross encoder filtering to enable exhaustive yet efficient processing. Across comparisons with graph-based, multimodal, and naive RAG baselines, existing approaches struggle to exceed 40% statementwise F1, while PluriHopRAG achieves substantial relative gains (18–52%) depending on the base LLM, highlighting the value of exhaustive retrieval and early filtering for pluri-hop QA. Collectively, the dataset and method reveal limitations of current QA systems on repetitive, distractor-rich corpora and point to a promising direction for robust, recall-sensitive retrieval strategies in real-world document collections.

Abstract

Recent advances in large language models (LLMs) and retrieval-augmented generation (RAG) have enabled progress on question answering (QA) when relevant evidence is in one (single-hop) or multiple (multi-hop) passages. Yet many realistic questions about recurring report data - medical records, compliance filings, maintenance logs - require aggregation across all documents, with no clear stopping point for retrieval and high sensitivity to even one missed passage. We term these pluri-hop questions and formalize them by three criteria: recall sensitivity, exhaustiveness, and exactness. To study this setting, we introduce PluriHopWIND, a diagnostic multilingual dataset of 48 pluri-hop questions built from 191 real-world wind industry reports in German and English. We show that PluriHopWIND is 8-40% more repetitive than other common datasets and thus has higher density of distractor documents, better reflecting practical challenges of recurring report corpora. We test a traditional RAG pipeline as well as graph-based and multimodal variants, and find that none of the tested approaches exceed 40% in statement-wise F1 score. Motivated by this, we propose PluriHopRAG, a RAG architecture that follows a "check all documents individually, filter cheaply" approach: it (i) decomposes queries into document-level subquestions and (ii) uses a cross-encoder filter to discard irrelevant documents before costly LLM reasoning. We find that PluriHopRAG achieves relative F1 score improvements of 18-52% depending on base LLM. Despite its modest size, PluriHopWIND exposes the limitations of current QA systems on repetitive, distractor-rich corpora. PluriHopRAG's performance highlights the value of exhaustive retrieval and early filtering as a powerful alternative to top-k methods.
Paper Structure (24 sections, 2 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 24 sections, 2 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. QA Generation
  5. Document Analysis
  6. Model
  7. Overview
  8. Query decomposition
  9. Document filtering
  10. Experimental Setup
  11. Results
  12. Ablations
  13. Metadata filter
  14. Fine-tuned query decomposition
  15. Conclusion
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Common types of questions RAG systems are used for.
  • Figure 2: Typical report in the PluriHopWIND dataset
  • Figure 3: Chunk repetitiveness for PluriHopWIND, Loong, and MultiHopRAG datasets
  • Figure 4: Diagram of PluriHopRAG algorithm
  • Figure 5: Behavior of cross-encoder based document filter for the PluriHopWIND dataset. Left: Receiver Operator Characteristic (ROC) curve for filter at different filter thresholds $\tau$. Right: distribution of estimated document relevance for relevant and irrelevant documents.