Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Experiment in Retrofitting Competency Questions for Existing Ontologies

Reham Alharbi, Valentina Tamma, Floriana Grasso, Terry Payne

TL;DR

This paper introduces RETROFIT-CQs, a pipeline that automatically retrofits competency questions from existing ontologies by extracting RDF triples, generating natural-language CQs via multiple zero-shot prompts to LLMs (gpt-3.5-turbo, gpt-4, Llama-2-70b-chat), and filtering to remove duplicates and modelling-specific or narrative questions. The approach is evaluated empirically against CORAL's curated CQs across three ontologies, showing high recall (≈0.95+) but variable precision depending on the LLM and prompt type; ontology-developer feedback on a Solar System ontology confirms the generated CQs are largely usable and identifies additional CQs that align with design intent. The results suggest LLMs can effectively surface usable CQs from ontology vocabularies, facilitating ontology reuse and evaluation where CQs were not originally published, while highlighting challenges in linguistic alignment and fidelity to modelling choices. The work offers a practical method for generating CQ artefacts to support reuse, testing, and requirement specification, with future avenues including larger-scale validation, multi-triple CQ generation, and richer prompt templates to balance recall and precision.

Abstract

Competency Questions (CQs) are a form of ontology functional requirements expressed as natural language questions. Inspecting CQs together with the axioms in an ontology provides critical insights into the intended scope and applicability of the ontology. CQs also underpin a number of tasks in the development of ontologies e.g. ontology reuse, ontology testing, requirement specification, and the definition of patterns that implement such requirements. Although CQs are integral to the majority of ontology engineering methodologies, the practice of publishing CQs alongside the ontological artefacts is not widely observed by the community. In this context, we present an experiment in retrofitting CQs from existing ontologies. We propose RETROFIT-CQs, a method to extract candidate CQs directly from ontologies using Generative AI. In the paper we present the pipeline that facilitates the extraction of CQs by leveraging Large Language Models (LLMs) and we discuss its application to a number of existing ontologies.

An Experiment in Retrofitting Competency Questions for Existing Ontologies

TL;DR

This paper introduces RETROFIT-CQs, a pipeline that automatically retrofits competency questions from existing ontologies by extracting RDF triples, generating natural-language CQs via multiple zero-shot prompts to LLMs (gpt-3.5-turbo, gpt-4, Llama-2-70b-chat), and filtering to remove duplicates and modelling-specific or narrative questions. The approach is evaluated empirically against CORAL's curated CQs across three ontologies, showing high recall (≈0.95+) but variable precision depending on the LLM and prompt type; ontology-developer feedback on a Solar System ontology confirms the generated CQs are largely usable and identifies additional CQs that align with design intent. The results suggest LLMs can effectively surface usable CQs from ontology vocabularies, facilitating ontology reuse and evaluation where CQs were not originally published, while highlighting challenges in linguistic alignment and fidelity to modelling choices. The work offers a practical method for generating CQ artefacts to support reuse, testing, and requirement specification, with future avenues including larger-scale validation, multi-triple CQ generation, and richer prompt templates to balance recall and precision.

Abstract

Competency Questions (CQs) are a form of ontology functional requirements expressed as natural language questions. Inspecting CQs together with the axioms in an ontology provides critical insights into the intended scope and applicability of the ontology. CQs also underpin a number of tasks in the development of ontologies e.g. ontology reuse, ontology testing, requirement specification, and the definition of patterns that implement such requirements. Although CQs are integral to the majority of ontology engineering methodologies, the practice of publishing CQs alongside the ontological artefacts is not widely observed by the community. In this context, we present an experiment in retrofitting CQs from existing ontologies. We propose RETROFIT-CQs, a method to extract candidate CQs directly from ontologies using Generative AI. In the paper we present the pipeline that facilitates the extraction of CQs by leveraging Large Language Models (LLMs) and we discuss its application to a number of existing ontologies.
Paper Structure (13 sections, 1 figure, 4 tables, 1 algorithm)

This paper contains 13 sections, 1 figure, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. The RETROFIT-CQ pipeline
  4. Extracting Triples from Ontologies
  5. LLM-based Question Generation
  6. Question Filtration
  7. Implementation of the Method
  8. Empirical Evaluation
  9. Generating Candidate CQs from Repositories of Requirements
  10. Retrofitting CQ Results
  11. Ontology Developer-Led Evaluation
  12. Discussion
  13. Conclusions

Figures (1)

  • Figure 1: The RETROFIT-CQs architecture.