DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

TL;DR

This work introduces DeepAmbigQAGen, an automatic pipeline that generates knowledge-graph grounded, ambiguous multi-hop QA tasks by aligning a Wikidata KG with Wikipedia. The resulting DeepAmbigQA dataset contains 3,600 questions, half of which include name ambiguity requiring multiple reasoning branches, challenging current LLM-based QA systems to achieve complete answer recall. Experimental results with SOTA models show pronounced gaps in answer completeness and increased difficulty on ambiguous queries, underscoring the need for retrieval-rich, ambiguity-aware reasoning. The paper argues for stronger QA systems that integrate comprehensive evidence gathering and robust disambiguation to achieve answer completeness in real-world scenarios.

Abstract

Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as Which actor from the film Heat won at least one Academy Award?, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DeepAmbigQAGen, an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DeepAmbigQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness.

Figures (23)

• Figure 1: Comparison between DeepAmbigQA questions (bottom), SimpleQA (top left), and AmbigQA (top right). Takeaway: Unlike SimpleQA and AmbigQA, DeepAmbigQA questions require both name disambiguation and multi-hop reasoning over a large intermediate evidence set, presenting a significant challenge for current LLM-based QA systems.
• Figure 2: Illustration of the DeepAmbigQAGen algorithm. Beginning with a large collection of entities and relations, a complex query is constructed by iteratively sampling atomic entities and new operations. The resulting query is then converted into natural language by prompting an LLM.
• Figure 2: Key data statistics of DeepAmbigQA.
• Figure 3: Human evaluation results of DeepAmbigQA on question naturalness and answer correctness. Scores are averaged over three annotators.
• Figure 3: Error-type breakdown for various LLMs.