Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Documents to Dialogue: Building KG-RAG Enhanced AI Assistants

Manisha Mukherjee, Sungchul Kim, Xiang Chen, Dan Luo, Tong Yu, Tung Mai

TL;DR

Large enterprise document stores challenge LLMs due to restricted access and potential hallucinations. The authors present KG-RAG, an incremental knowledge-graph-based retrieval system that augments LLM prompts with high-confidence, provenance-traced facts extracted from enterprise documents. The approach achieves substantial gains in answer relevance ( Irrelevant down ~52%, Fully Relevant up ~88%) and strong semantic alignment (avg cosine similarity ~0.89) compared to a production baseline. This work demonstrates a practical pathway to robust, private-data QA in enterprise AI assistants and offers a framework adaptable to other organizations and data ecosystems.

Abstract

The Adobe Experience Platform AI Assistant is a conversational tool that enables organizations to interact seamlessly with proprietary enterprise data through a chatbot. However, due to access restrictions, Large Language Models (LLMs) cannot retrieve these internal documents, limiting their ability to generate accurate zero-shot responses. To overcome this limitation, we use a Retrieval-Augmented Generation (RAG) framework powered by a Knowledge Graph (KG) to retrieve relevant information from external knowledge sources, enabling LLMs to answer questions over private or previously unseen document collections. In this paper, we propose a novel approach for building a high-quality, low-noise KG. We apply several techniques, including incremental entity resolution using seed concepts, similarity-based filtering to deduplicate entries, assigning confidence scores to entity-relation pairs to filter for high-confidence pairs, and linking facts to source documents for provenance. Our KG-RAG system retrieves relevant tuples, which are added to the user prompts context before being sent to the LLM generating the response. Our evaluation demonstrates that this approach significantly enhances response relevance, reducing irrelevant answers by over 50% and increasing fully relevant answers by 88% compared to the existing production system.

From Documents to Dialogue: Building KG-RAG Enhanced AI Assistants

TL;DR

Large enterprise document stores challenge LLMs due to restricted access and potential hallucinations. The authors present KG-RAG, an incremental knowledge-graph-based retrieval system that augments LLM prompts with high-confidence, provenance-traced facts extracted from enterprise documents. The approach achieves substantial gains in answer relevance ( Irrelevant down ~52%, Fully Relevant up ~88%) and strong semantic alignment (avg cosine similarity ~0.89) compared to a production baseline. This work demonstrates a practical pathway to robust, private-data QA in enterprise AI assistants and offers a framework adaptable to other organizations and data ecosystems.

Abstract

The Adobe Experience Platform AI Assistant is a conversational tool that enables organizations to interact seamlessly with proprietary enterprise data through a chatbot. However, due to access restrictions, Large Language Models (LLMs) cannot retrieve these internal documents, limiting their ability to generate accurate zero-shot responses. To overcome this limitation, we use a Retrieval-Augmented Generation (RAG) framework powered by a Knowledge Graph (KG) to retrieve relevant information from external knowledge sources, enabling LLMs to answer questions over private or previously unseen document collections. In this paper, we propose a novel approach for building a high-quality, low-noise KG. We apply several techniques, including incremental entity resolution using seed concepts, similarity-based filtering to deduplicate entries, assigning confidence scores to entity-relation pairs to filter for high-confidence pairs, and linking facts to source documents for provenance. Our KG-RAG system retrieves relevant tuples, which are added to the user prompts context before being sent to the LLM generating the response. Our evaluation demonstrates that this approach significantly enhances response relevance, reducing irrelevant answers by over 50% and increasing fully relevant answers by 88% compared to the existing production system.

Paper Structure

This paper contains 13 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Implementation
  3. Document cleaning and summarization
  4. Seed Concept, Entity Discovery and Deduplication
  5. Relation Extraction and Confidence Scoring
  6. Provenance
  7. Evaluation
  8. Dataset
  9. LLM as Judge
  10. Cosine Similarity for Semantic Matching
  11. Results and Discussion
  12. Related Work
  13. Conclusion

Figures (4)

  • Figure 1: Example of AI assistant generating a response saying that it is out of scope
  • Figure 2: End-to-end pipeline for Knowledge Graph-based Retrieval-Augmented Generation (KG-RAG).
  • Figure 3: Evaluation framework for response relevance. The pipeline compares baseline production answers with KG-RAG-enhanced responses.
  • Figure 4: LLM-as-Judge KDE plot showing the distribution of relevance scores for baseline production and KG-RAG.