Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt-Time Ontology-Driven Symbolic Knowledge Capture with Large Language Models

Tolga Çöplü, Arto Bendiken, Andrii Skomorokhov, Eduard Bateiko, Stephen Cobb

TL;DR

The paper tackles enabling LLMs to learn personal information from user prompts by leveraging ontology-driven symbolic knowledge capture at prompt-time. It compares in-context learning with a fine-tuning approach and demonstrates that fine-tuning on a predefined ontology (subset of KNOW) efficiently teaches the model to extract and populate subject-predicate-object triples. A dataset of 175 prompts (143 training/test plus 32 generic) serialized in Turtle, focusing on 12 family-related concepts, shows that using eight prompts per concept and 18 fine-tuning epochs yields robust precision, recall, and F1 in knowledge capture. The work highlights feasibility and scalability of combining neural models with symbolic ontologies for personalized AI, with future work aimed at integrating the generated knowledge graph into knowledge utilization by the LLM and providing code and data publicly.

Abstract

In applications such as personal assistants, large language models (LLMs) must consider the user's personal information and preferences. However, LLMs lack the inherent ability to learn from user interactions. This paper explores capturing personal information from user prompts using ontology and knowledge-graph approaches. We use a subset of the KNOW ontology, which models personal information, to train the language model on these concepts. We then evaluate the success of knowledge capture using a specially constructed dataset. Our code and datasets are publicly available at https://github.com/HaltiaAI/paper-PTODSKC

Prompt-Time Ontology-Driven Symbolic Knowledge Capture with Large Language Models

TL;DR

The paper tackles enabling LLMs to learn personal information from user prompts by leveraging ontology-driven symbolic knowledge capture at prompt-time. It compares in-context learning with a fine-tuning approach and demonstrates that fine-tuning on a predefined ontology (subset of KNOW) efficiently teaches the model to extract and populate subject-predicate-object triples. A dataset of 175 prompts (143 training/test plus 32 generic) serialized in Turtle, focusing on 12 family-related concepts, shows that using eight prompts per concept and 18 fine-tuning epochs yields robust precision, recall, and F1 in knowledge capture. The work highlights feasibility and scalability of combining neural models with symbolic ontologies for personalized AI, with future work aimed at integrating the generated knowledge graph into knowledge utilization by the LLM and providing code and data publicly.

Abstract

In applications such as personal assistants, large language models (LLMs) must consider the user's personal information and preferences. However, LLMs lack the inherent ability to learn from user interactions. This paper explores capturing personal information from user prompts using ontology and knowledge-graph approaches. We use a subset of the KNOW ontology, which models personal information, to train the language model on these concepts. We then evaluate the success of knowledge capture using a specially constructed dataset. Our code and datasets are publicly available at https://github.com/HaltiaAI/paper-PTODSKC
Paper Structure (10 sections, 5 figures)

This paper contains 10 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Ontology-Driven Symbolic Knowledge Capture
  3. Ontology-Driven Knowledge Capture with Fine-Tuning
  4. Experimental Setup
  5. Development Framework
  6. Language Model
  7. Predefined Ontology
  8. Dataset
  9. Performance Evaluation
  10. Conclusion

Figures (5)

  • Figure 1: A visual representation of the ontology design used in this paper.
  • Figure 2: Occurrences of ontology concepts in the prepared dataset.
  • Figure 3: Occurrences of ontology concepts in the test dataset.
  • Figure 4: Ontology population performance (precision, recall, and f1-score) for various fine-tuning dataset sizes.
  • Figure 5: Ontology population performance (precision, recall, and f1-score) of 8 samples per ontology concept for various epochs.