Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KnowGPT: Knowledge Graph based Prompting for Large Language Models

Qinggang Zhang, Junnan Dong, Hao Chen, Daochen Zha, Zailiang Yu, Xiao Huang

TL;DR

The paper tackles hallucinations in large language models by grounding responses in structured domain knowledge from knowledge graphs. It proposes KnowGPT, a two-stage framework that first learns to extract informative, concise KG reasoning traces through deep reinforcement learning, and then automatically constructs effective prompts with a multi-armed bandit that balances extraction strategy and prompt format. Empirical results on CommonsenseQA, OpenBookQA, and MedQA show KnowGPT consistently outperforms baselines, including GPT-3.5 and GPT-4, with a notable 92.6% accuracy on the OpenBookQA leaderboard, nearing human performance. The approach offers a practical, cost-efficient path to ground LLMs via API-based prompting without requiring white-box access to model internals. Limitations include noisy KG data and opportunities for KG refinement and broader domain applicability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in many real-world applications. Nonetheless, LLMs are often criticized for their tendency to produce hallucinations, wherein the models fabricate incorrect statements on tasks beyond their knowledge and perception. To alleviate this issue, researchers have explored leveraging the factual knowledge in knowledge graphs (KGs) to ground the LLM's responses in established facts and principles. However, most state-of-the-art LLMs are closed-source, making it challenging to develop a prompting framework that can efficiently and effectively integrate KGs into LLMs with hard prompts only. Generally, existing KG-enhanced LLMs usually suffer from three critical issues, including huge search space, high API costs, and laborious prompt engineering, that impede their widespread application in practice. To this end, we introduce a novel Knowledge Graph based PrompTing framework, namely KnowGPT, to enhance LLMs with domain knowledge. KnowGPT contains a knowledge extraction module to extract the most informative knowledge from KGs, and a context-aware prompt construction module to automatically convert extracted knowledge into effective prompts. Experiments on three benchmarks demonstrate that KnowGPT significantly outperforms all competitors. Notably, KnowGPT achieves a 92.6% accuracy on OpenbookQA leaderboard, comparable to human-level performance.

KnowGPT: Knowledge Graph based Prompting for Large Language Models

TL;DR

The paper tackles hallucinations in large language models by grounding responses in structured domain knowledge from knowledge graphs. It proposes KnowGPT, a two-stage framework that first learns to extract informative, concise KG reasoning traces through deep reinforcement learning, and then automatically constructs effective prompts with a multi-armed bandit that balances extraction strategy and prompt format. Empirical results on CommonsenseQA, OpenBookQA, and MedQA show KnowGPT consistently outperforms baselines, including GPT-3.5 and GPT-4, with a notable 92.6% accuracy on the OpenBookQA leaderboard, nearing human performance. The approach offers a practical, cost-efficient path to ground LLMs via API-based prompting without requiring white-box access to model internals. Limitations include noisy KG data and opportunities for KG refinement and broader domain applicability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in many real-world applications. Nonetheless, LLMs are often criticized for their tendency to produce hallucinations, wherein the models fabricate incorrect statements on tasks beyond their knowledge and perception. To alleviate this issue, researchers have explored leveraging the factual knowledge in knowledge graphs (KGs) to ground the LLM's responses in established facts and principles. However, most state-of-the-art LLMs are closed-source, making it challenging to develop a prompting framework that can efficiently and effectively integrate KGs into LLMs with hard prompts only. Generally, existing KG-enhanced LLMs usually suffer from three critical issues, including huge search space, high API costs, and laborious prompt engineering, that impede their widespread application in practice. To this end, we introduce a novel Knowledge Graph based PrompTing framework, namely KnowGPT, to enhance LLMs with domain knowledge. KnowGPT contains a knowledge extraction module to extract the most informative knowledge from KGs, and a context-aware prompt construction module to automatically convert extracted knowledge into effective prompts. Experiments on three benchmarks demonstrate that KnowGPT significantly outperforms all competitors. Notably, KnowGPT achieves a 92.6% accuracy on OpenbookQA leaderboard, comparable to human-level performance.
Paper Structure (33 sections, 6 equations, 3 figures, 11 tables)

This paper contains 33 sections, 6 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. KnowGPT Framework
  4. Knowledge Extraction with Deep Reinforcement Learning
  5. Context-relatedness Auxiliary Reward
  6. Conciseness Auxiliary Reward
  7. Training Policy Network
  8. Prompt Construction with Multi-armed Bandit
  9. Implementation
  10. Experiments
  11. Experimental Setup
  12. Main Results (RQ1)
  13. Leaderboard Ranking
  14. Ablation Studies (RQ2)
  15. Limitation
  16. ...and 18 more sections

Figures (3)

  • Figure 1: A real-world question from OpenbookQA. GPT-3.5 could effectively correct the answer given the scientific reasoning background from ConceptNet (blue: question concepts, red: answers, grey: entities not present in questions).
  • Figure 2: The overall architecture of our proposed knowledge graph prompting framework, i.e., KnowGPT. Given the question context with multiple choices, we first retrieve a question-specific subgraph from the real-world KG. Knowledge Extraction is first dedicated to searching for the most informative and concise reasoning background subject to the context. Then the Prompt Construction module is optimized to prioritize the combination of knowledge and formats subject to the given question.
  • Figure 3: A case study on exploring the effectiveness of different prompt formats for particular questions. The extracted knowledge is shown in the middle of this figure in the form of a graph, where the nodes in blue are the key topic entities and the red is the target answer. The text boxes at the bottom are the final prompts generated based on three different formats.