EffiQA: Efficient Question-Answering with Strategic Multi-Model Collaboration on Knowledge Graphs

TL;DR

EffiQA leverages the commonsense capability of LLMs to explore potential reasoning pathways through global planning and offloads semantic pruning to a small plug-in model for efficient KG exploration to improve the global planning and efficient KG exploration.

Abstract

While large language models (LLMs) have shown remarkable capabilities in natural language processing, they struggle with complex, multi-step reasoning tasks involving knowledge graphs (KGs). Existing approaches that integrate LLMs and KGs either underutilize the reasoning abilities of LLMs or suffer from prohibitive computational costs due to tight coupling. To address these limitations, we propose a novel collaborative framework named EffiQA that can strike a balance between performance and efficiency via an iterative paradigm. EffiQA consists of three stages: global planning, efficient KG exploration, and self-reflection. Specifically, EffiQA leverages the commonsense capability of LLMs to explore potential reasoning pathways through global planning. Then, it offloads semantic pruning to a small plug-in model for efficient KG exploration. Finally, the exploration results are fed to LLMs for self-reflection to further improve the global planning and efficient KG exploration. Empirical evidence on multiple KBQA benchmarks shows EffiQA's effectiveness, achieving an optimal balance between reasoning accuracy and computational costs. We hope the proposed new framework will pave the way for efficient, knowledge-intensive querying by redefining the integration of LLMs and KGs, fostering future research on knowledge-based question answering.

Figures (7)

• Figure 1: LLM-based KBQA includes three paradigms: loose-coupling (LLM-only with prompts), tight-coupling (LLM exploring KG iteratively), and moderate-coupling (LLM for planning, plug-in model for KG exploration).
• Figure 2: The EffiQA workflow consists of three stages. First, the LLM decomposes the problem and generates instructions that include simulated answers and actions based on the problem’s logic. Next, EffiQA employs a plug-in model to execute these instructions, perform efficient knowledge graph exploration, and identify potential issues. Finally, the LLM reviews the identified problems, iteratively replans, and produces answers once sufficient information is available.
• Figure 3: Despite the challenges associated with the temporal constraints of the training corpus (up to March 2021), the plug-in model effectively executes accurate match pruning. The detailed entity typing depicted in the figure serves solely for illustrative purposes
• Figure 4: Accuracy by Reflection Iterations on WebQSP Dataset (MAX_REFLECTION=5)
• Figure 5: Path Recall Rate and Cost Efficiency of various model sizes on CWQ and WebQSP datasets. The medium-sized entity typing model achieves an optimal balance between recall and cost. $\alpha$: RoBERTa models fine-tuned with increased parameters. $\beta$: Activation parameters calculated using the MoE model. $\gamma$: GPT-4 parameter estimates based on scale, as exact numbers are undisclosed.