EWEK-QA: Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systems

TL;DR

EWEK-QA introduces a dual-knowledge grounding framework for citation-based QA that combines an adaptive web retriever with a KG-driven sub-graph extractor (ToG-E) to deliver accurate, cite-grounded answers while preserving efficiency on open-domain and multi-hop tasks. It foregrounds an adaptive Web-Quote extraction pipeline (PS+EE) and a KG sub-graph retrieval method that avoid extensive LLM calls, culminating in an answer composer based on open-source LLMs. Across KGQA and ODQA benchmarks, EWEK-QA outperforms web-only and KG-only baselines and achieves substantial speedups (3×–6×) over ToG, while maintaining high citation accuracy (~89.6%). The results demonstrate the value of integrating web and KG knowledge sources for scalable, grounded QA in practical settings, with strong human-evaluation support for its improvements in accuracy and citation quality.

Abstract

The emerging citation-based QA systems are gaining more attention especially in generative AI search applications. The importance of extracted knowledge provided to these systems is vital from both accuracy (completeness of information) and efficiency (extracting the information in a timely manner). In this regard, citation-based QA systems are suffering from two shortcomings. First, they usually rely only on web as a source of extracted knowledge and adding other external knowledge sources can hamper the efficiency of the system. Second, web-retrieved contents are usually obtained by some simple heuristics such as fixed length or breakpoints which might lead to splitting information into pieces. To mitigate these issues, we propose our enhanced web and efficient knowledge graph (KG) retrieval solution (EWEK-QA) to enrich the content of the extracted knowledge fed to the system. This has been done through designing an adaptive web retriever and incorporating KGs triples in an efficient manner. We demonstrate the effectiveness of EWEK-QA over the open-source state-of-the-art (SoTA) web-based and KG baseline models using a comprehensive set of quantitative and human evaluation experiments. Our model is able to: first, improve the web-retriever baseline in terms of extracting more relevant passages (>20\%), the coverage of answer span (>25\%) and self containment (>35\%); second, obtain and integrate KG triples into its pipeline very efficiently (by avoiding any LLM calls) to outperform the web-only and KG-only SoTA baselines significantly in 7 quantitative QA tasks and our human evaluation.

Figures (6)

• Figure 1: Overview of performance vs. efficiency of EWEK-QA (KG+Web), WebGLM (Web-only), and ToG (KG-only) on the WebQSP dataset (See Table \ref{['tab:efficiency']} for details). Each circle represents one solution with its LLM's name and the number of calls to the LLM (indicated as $\times n$). The size of each circle indicates the relative size of its corresponding backbone LLM. The relative speed represents the speed with respect to ToG with LLaMA-2-13B. Bear in mind that ToG with ChatGPT needs to call the closed-source ChatGPT system 8 times on average, which can increase the expenses and also raise privacy concerns for sensitive applications.
• Figure 2: Comparison of EWEK-QA , ToG tog, and WebGLM webglm pipelines. EWEK-QA utilizes both knowledge modalities which enables to correctly answer both question types using a single LLM call. ToG requires $O(ND)$ costly calls where $N$ and $D$ represent the beam search width and depth respectively. WebGLM relies solely on the web which makes it unfit for multi-hop reasoning questions.
• Figure 3: Sources of the quotes cited by the Answer Composer across queries from two KGQA and two ODQA datasets. "None" denotes that the answer contains no citations. "Web Quotes + KG Triples" indicate that both KG Triples and Web Quotes are cited.
• Figure 4: This figure depicts what percentage of the quotes provided by our knowledge extraction approach are cited in the final answer. Note that Paragraph Splitter and Evidence Extractor correspond to Web Quotes, and KG Triples come from the knowledge graph.
• Figure 5: The pipeline for our adaptive web retrieval module.