Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalizing Large Language Models using Retrieval Augmented Generation and Knowledge Graph

Deeksha Prahlad, Chanhee Lee, Dongha Kim, Hokeun Kim

TL;DR

The paper tackles hallucinations in LLM outputs by pairing retrieval-augmented generation with knowledge graphs to deliver personalized, timely information, focusing on calendar data and on-device deployment with smaller Llama-2-Chat variants. It introduces a complete pipeline from synthetic dataset creation to KG construction, embedding-based retrieval, and prompt-driven response generation, comparing KG-enhanced retrieval to a baseline RAG setup. Empirical results show significant improvements in ROUGE and BLEU metrics and reduced latency, demonstrating the value of structured, up-to-date personal data stored as KGs for privacy-preserving personalization. The work highlights the practicality of on-device personalization, enabling smarter responses without sending sensitive data to cloud LLM providers, and it provides publicly available datasets and code to foster further evaluation and adoption.

Abstract

The advent of large language models (LLMs) has allowed numerous applications, including the generation of queried responses, to be leveraged in chatbots and other conversational assistants. Being trained on a plethora of data, LLMs often undergo high levels of over-fitting, resulting in the generation of extra and incorrect data, thus causing hallucinations in output generation. One of the root causes of such problems is the lack of timely, factual, and personalized information fed to the LLM. In this paper, we propose an approach to address these problems by introducing retrieval augmented generation (RAG) using knowledge graphs (KGs) to assist the LLM in personalized response generation tailored to the users. KGs have the advantage of storing continuously updated factual information in a structured way. While our KGs can be used for a variety of frequently updated personal data, such as calendar, contact, and location data, we focus on calendar data in this paper. Our experimental results show that our approach works significantly better in understanding personal information and generating accurate responses compared to the baseline LLMs using personal data as text inputs, with a moderate reduction in response time.

Personalizing Large Language Models using Retrieval Augmented Generation and Knowledge Graph

TL;DR

The paper tackles hallucinations in LLM outputs by pairing retrieval-augmented generation with knowledge graphs to deliver personalized, timely information, focusing on calendar data and on-device deployment with smaller Llama-2-Chat variants. It introduces a complete pipeline from synthetic dataset creation to KG construction, embedding-based retrieval, and prompt-driven response generation, comparing KG-enhanced retrieval to a baseline RAG setup. Empirical results show significant improvements in ROUGE and BLEU metrics and reduced latency, demonstrating the value of structured, up-to-date personal data stored as KGs for privacy-preserving personalization. The work highlights the practicality of on-device personalization, enabling smarter responses without sending sensitive data to cloud LLM providers, and it provides publicly available datasets and code to foster further evaluation and adoption.

Abstract

The advent of large language models (LLMs) has allowed numerous applications, including the generation of queried responses, to be leveraged in chatbots and other conversational assistants. Being trained on a plethora of data, LLMs often undergo high levels of over-fitting, resulting in the generation of extra and incorrect data, thus causing hallucinations in output generation. One of the root causes of such problems is the lack of timely, factual, and personalized information fed to the LLM. In this paper, we propose an approach to address these problems by introducing retrieval augmented generation (RAG) using knowledge graphs (KGs) to assist the LLM in personalized response generation tailored to the users. KGs have the advantage of storing continuously updated factual information in a structured way. While our KGs can be used for a variety of frequently updated personal data, such as calendar, contact, and location data, we focus on calendar data in this paper. Our experimental results show that our approach works significantly better in understanding personal information and generating accurate responses compared to the baseline LLMs using personal data as text inputs, with a moderate reduction in response time.
Paper Structure (13 sections, 4 figures, 3 tables)

This paper contains 13 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Approach
  3. Overall Workflow
  4. Dataset Generation
  5. Knowledge Graphs
  6. Response Generation
  7. Dataset Generation
  8. Evaluation
  9. Baseline
  10. Experimental Setup
  11. Evaluation Criteria
  12. Experimental Results
  13. Conclusion

Figures (4)

  • Figure 1: Overall workflow of the proposed approach
  • Figure 2: Knowledge graph for a sample of the calendar data from our dataset.
  • Figure 3: Experimental results comparing three types of ROUGE metric scores (ROUGE-1, ROUGE-2, and ROUGE-L) of three Llama-2-Chat models (7B, 13B, and 70B) between the baseline and our approach.
  • Figure 4: Experimental results comparing BLEU scores of Llama-2-Chat models (7B, 13B, and 70B) between the baseline and our approach.