Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable Question Answering with Knowledge Graphs

Kartikeya Aneja, Manasvi Srivastava, Subhayan Das, Nagender Aneja

TL;DR

The paper addresses the limitations of retrieval-augmented generation (RAG) QA systems, such as hallucinations and opacity, by proposing a knowledge-graph–based QA pipeline that first generates QA pairs from documents and builds a structured graph, then retrieves subgraphs using embeddings and fuzzy matching before paraphrasing and reranking to produce final answers. The approach emphasizes interpretability and traceability by operating on a KG rather than unstructured text chunks, and it demonstrates competitive accuracy on CRAG (71.9% with Llama-3.2 and 54.4% with GPT-3.5-Turbo) with robustness to question perturbations. Key contributions include a two-stage KG construction and retrieval workflow, the use of LLMGraphTransformer for structured extraction, and a paraphrase-plus-rerank finalization step that yields fluent, verifiable outputs. The results suggest practical benefits for knowledge-intensive domains and offer a transparent alternative to black-box RAG, with code and datasets publicly available for reproducibility.

Abstract

This paper presents a question answering system that operates exclusively on a knowledge graph retrieval without relying on retrieval augmented generation (RAG) with large language models (LLMs). Instead, a small paraphraser model is used to paraphrase the entity relationship edges retrieved from querying the knowledge graph. The proposed pipeline is divided into two main stages. The first stage involves pre-processing a document to generate sets of question-answer (QA) pairs. The second stage converts these QAs into a knowledge graph from which graph-based retrieval is performed using embeddings and fuzzy techniques. The graph is queried, re-ranked, and paraphrased to generate a final answer. This work includes an evaluation using LLM-as-a-judge on the CRAG benchmark, which resulted in accuracies of 71.9% and 54.4% using LLAMA-3.2 and GPT-3.5-Turbo, respectively.

Interpretable Question Answering with Knowledge Graphs

TL;DR

The paper addresses the limitations of retrieval-augmented generation (RAG) QA systems, such as hallucinations and opacity, by proposing a knowledge-graph–based QA pipeline that first generates QA pairs from documents and builds a structured graph, then retrieves subgraphs using embeddings and fuzzy matching before paraphrasing and reranking to produce final answers. The approach emphasizes interpretability and traceability by operating on a KG rather than unstructured text chunks, and it demonstrates competitive accuracy on CRAG (71.9% with Llama-3.2 and 54.4% with GPT-3.5-Turbo) with robustness to question perturbations. Key contributions include a two-stage KG construction and retrieval workflow, the use of LLMGraphTransformer for structured extraction, and a paraphrase-plus-rerank finalization step that yields fluent, verifiable outputs. The results suggest practical benefits for knowledge-intensive domains and offer a transparent alternative to black-box RAG, with code and datasets publicly available for reproducibility.

Abstract

This paper presents a question answering system that operates exclusively on a knowledge graph retrieval without relying on retrieval augmented generation (RAG) with large language models (LLMs). Instead, a small paraphraser model is used to paraphrase the entity relationship edges retrieved from querying the knowledge graph. The proposed pipeline is divided into two main stages. The first stage involves pre-processing a document to generate sets of question-answer (QA) pairs. The second stage converts these QAs into a knowledge graph from which graph-based retrieval is performed using embeddings and fuzzy techniques. The graph is queried, re-ranked, and paraphrased to generate a final answer. This work includes an evaluation using LLM-as-a-judge on the CRAG benchmark, which resulted in accuracies of 71.9% and 54.4% using LLAMA-3.2 and GPT-3.5-Turbo, respectively.
Paper Structure (13 sections, 3 figures, 2 tables)

This paper contains 13 sections, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. RESEARCH METHODOLOGY
  4. QA Generation from Document
  5. Knowledge Graph Creation
  6. Knowledge Graph Construction
  7. Retrieval Phase
  8. Paraphrase Phase
  9. Reranker Phase
  10. EVALUATION METHODOLOGY AND RESULTS
  11. LLM-as-a-Judge Evaluation
  12. Robustness Evaluation via Question Perturbation
  13. CONCLUSION

Figures (3)

  • Figure 1: LLM Graph Transformer setup
  • Figure 2: Prompt used for LLM-as-a-judge
  • Figure 3: Perturbation Sample