A RAG Approach for Generating Competency Questions in Ontology Engineering

TL;DR

The paper tackles the slow, expert-intensive task of crafting competency questions by introducing a retrieval-augmented generation (RAG) approach that uses domain literature as a knowledge base to steer LLMs in CQ generation. It deploys a three-step RAG pipeline (indexing, retrieval, generation) and a zero-shot prompt template to produce CQs for two domain ontology tasks, KG-EmpiRE and HCIO, while exploring the impact of $N_{paper}$ and $temp$ on performance using $p$-based precision and consistency metrics. Results show that RAG improves CQ generation for the more domain-specific KG-EmpiRE, with precision generally increasing as the knowledge base grows, whereas HCIO favors zero-shot prompting, and temperature has little effect on consistency. These findings suggest that domain knowledge augmentation with RAG can accelerate CQ creation in ontology engineering, particularly for concrete domains, and motivate future work on additional tasks and open-source LLMs to manage costs.

Abstract

Competency question (CQ) formulation is central to several ontology development and evaluation methodologies. Traditionally, the task of crafting these competency questions heavily relies on the effort of domain experts and knowledge engineers which is often time-consuming and labor-intensive. With the emergence of Large Language Models (LLMs), there arises the possibility to automate and enhance this process. Unlike other similar works which use existing ontologies or knowledge graphs as input to LLMs, we present a retrieval-augmented generation (RAG) approach that uses LLMs for the automatic generation of CQs given a set of scientific papers considered to be a domain knowledge base. We investigate its performance and specifically, we study the impact of different number of papers to the RAG and different temperature setting of the LLM. We conduct experiments using GPT-4 on two domain ontology engineering tasks and compare results against ground-truth CQs constructed by domain experts. Empirical assessments on the results, utilizing evaluation metrics (precision and consistency), reveal that compared to zero-shot prompting, adding relevant domain knowledge to the RAG improves the performance of LLMs on generating CQs for concrete ontology engineering tasks.

Figures (3)

• Figure 1: RAG pipeline with three hyperparameters: $N_{paper}, temp$ and $model$
• Figure 2: Precision of using RAG with gpt-4o to generate CQs, compared to zero-shot prompting
• Figure 3: Standard deviation of precision for the task performance and standard deviation of cosine similarity for generated text with different temperature.