Can Large Language Models Understand DL-Lite Ontologies? An Empirical Study

TL;DR

This work empirically analyze the LLMs' capability of understanding DL-Lite ontologies covering 6 representative tasks from syntactic and semantic aspects and finds that LLMs can understand formal syntax and model-theoretic semantics of concepts and roles.

Abstract

Large language models (LLMs) have shown significant achievements in solving a wide range of tasks. Recently, LLMs' capability to store, retrieve and infer with symbolic knowledge has drawn a great deal of attention, showing their potential to understand structured information. However, it is not yet known whether LLMs can understand Description Logic (DL) ontologies. In this work, we empirically analyze the LLMs' capability of understanding DL-Lite ontologies covering 6 representative tasks from syntactic and semantic aspects. With extensive experiments, we demonstrate both the effectiveness and limitations of LLMs in understanding DL-Lite ontologies. We find that LLMs can understand formal syntax and model-theoretic semantics of concepts and roles. However, LLMs struggle with understanding TBox NI transitivity and handling ontologies with large ABoxes. We hope that our experiments and analyses provide more insights into LLMs and inspire to build more faithful knowledge engineering solutions.

Figures (8)

• Figure 1: Illustration and examples of evaluation tasks.
• Figure 2: Evaluation pipeline for syntax checking, subsumption of concepts or roles, and instance checking.
• Figure 3: Performances of LLMs in subsumption of concepts or roles and instance checking.
• Figure 4: Performances for case study of tranitivity rules.
• Figure 5: Performances for probing of inverse role property and (inverse) functional role property.