HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions

TL;DR

HopWeaver presents the first fully automatic cross-document framework for synthesizing authentic multi-hop questions (bridge and comparison) directly from raw corpora. It combines a novel bridge- and comparison-question synthesis pipeline with a fine-tuned reranker and a polishing/validation module, enabling cost-effective generation of high-quality MHQA datasets. A comprehensive, multi-dimensional evaluation framework, including LLM-based judging and retrieval-quality metrics, demonstrates that HopWeaver-produced questions can match or exceed human benchmarks while offering substantial scalability and domain adaptability. End-to-end RAG evaluations reveal that the synthesized data challenge current systems in meaningful new ways, exposing limitations and guiding targeted improvements. The work provides a practical tool for constructing challenging benchmarks and training data, particularly in domains with limited annotated resources.

Abstract

Multi-Hop Question Answering (MHQA) is crucial for evaluating the model's capability to integrate information from diverse sources. However, creating extensive and high-quality MHQA datasets is challenging: (i) manual annotation is expensive, and (ii) current synthesis methods often produce simplistic questions or require extensive manual guidance. This paper introduces HopWeaver, the first cross-document framework synthesizing authentic multi-hop questions without human intervention. HopWeaver synthesizes bridge and comparison questions through an innovative pipeline that identifies complementary documents and constructs authentic reasoning paths to ensure true multi-hop reasoning. We further present a comprehensive system for evaluating the synthesized multi-hop questions. Empirical evaluations demonstrate that the synthesized questions achieve comparable or superior quality to human-annotated datasets at a lower cost. Our framework provides a valuable tool for the research community: it can automatically generate challenging benchmarks from any raw corpus, which opens new avenues for both evaluation and targeted training to improve the reasoning capabilities of advanced QA models, especially in domains with scarce resources.

Figures (9)

• Figure 1: Examples of two multi-hop questions synthesized by HopWeaver: Bridge (top) and Comparison (bottom) question. These involve cross-document reasoning via a bridge entity or a shared attribute.
• Figure 2: HopWeaver: Question Synthesis Framework
• Figure 3: Fine-Tuning Reranker
• Figure 4: Multi-dimensional quality evaluation to compare HopWeaver-synthesized questions (by different LLMs) against baseline datasets across key criteria, with scale truncated to [3.0, 4.5] for visualization clarity (full scale: [1, 5]).
• Figure 5: An example of a bridge question synthesized by HopWeaver. The figure illustrates the source documents, the identified bridge entity, the reasoning steps, and the final generated question-answer pair.