OntoChat: a Framework for Conversational Ontology Engineering using Language Models

TL;DR

OntoChat addresses the challenges of ontology engineering in multi-stakeholder, large-scale projects by combining conversational elicitation with LLM-assisted analysis and testing. The framework supports creating user stories, extracting and filtering competency questions, clustering CQs, and performing early ontology testing through textual verbalisation, avoiding SPARQL-heavy workflows. The authors validate the approach by replicating the Music Meta ontology's development and reporting positive feedback from domain experts and ontology engineers, plus promising testing accuracy (87.5%). The work offers an open-source, end-to-end tool for conversational OE and points to future work integrating domain-specific safeguards against hallucinations and biases while improving cost and timeline estimates.

Abstract

Ontology engineering (OE) in large projects poses a number of challenges arising from the heterogeneous backgrounds of the various stakeholders, domain experts, and their complex interactions with ontology designers. This multi-party interaction often creates systematic ambiguities and biases from the elicitation of ontology requirements, which directly affect the design, evaluation and may jeopardise the target reuse. Meanwhile, current OE methodologies strongly rely on manual activities (e.g., interviews, discussion pages). After collecting evidence on the most crucial OE activities, we introduce \textbf{OntoChat}, a framework for conversational ontology engineering that supports requirement elicitation, analysis, and testing. By interacting with a conversational agent, users can steer the creation of user stories and the extraction of competency questions, while receiving computational support to analyse the overall requirements and test early versions of the resulting ontologies. We evaluate OntoChat by replicating the engineering of the Music Meta Ontology, and collecting preliminary metrics on the effectiveness of each component from users. We release all code at https://github.com/King-s-Knowledge-Graph-Lab/OntoChat.

Figures (11)

• Figure 1: Illustration of the workflow alongside the main features in OntoChat.
• Figure 2: Responses from the Ontology Engineering survey to address RQ1. Replies quantify the agreement of participants with respect to each statement on a 5-point Likert scale, where 1 (absolutely disagree) to 5 (absolutely agree), with 3 being a neutral response (neither agree nor disagree).
• Figure 3: User ratings on the Collaborative User Story Creation functionality of OntoChat. The evaluation was performed by $N=6$ domain experts.
• Figure 4: User ratings on the Competency Question Extraction functionality of OntoChat. This evaluation was performed by $N=8$ ontology engineers.
• Figure 5: User ratings on the Competency Question Clustering functionality performed by $N=8$ ontology engineers.