OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learning

TL;DR

OntoURL addresses the gap in evaluating large language models on structured symbolic knowledge by introducing a taxonomy of ontological capabilities (understanding, reasoning, learning) and a comprehensive benchmark built from 40 ontologies across 8 domains. It comprises 57,303 questions over 15 tasks to quantify LLM performance on three capabilities, with zero-shot and few-shot prompts and human evaluation. The study finds that current open-source LLMs excel at ontological understanding but struggle with reasoning over implicit relations and with autonomous ontological learning, though larger models and certain prompting strategies can mitigate some weaknesses. OntoURL serves as a critical resource for benchmarking and guiding the integration of LLMs with formal knowledge representations, while acknowledging domain coverage and language scope limitations as avenues for future work.

Abstract

Large language models have demonstrated remarkable capabilities across a wide range of tasks, yet their ability to process structured symbolic knowledge remains underexplored. To address this gap, we propose a taxonomy of ontological capabilities and introduce OntoURL, the first comprehensive benchmark designed to systematically evaluate LLMs' capabilities in handling ontologies -- formal and symbolic representations of domain knowledge. Based on the proposed taxonomy, OntoURL systematically assesses three dimensions: understanding, reasoning, and learning through 15 distinct tasks comprising 57,303 questions derived from 40 ontologies across 8 domains. Experiments with 20 open-source LLMs reveal significant performance differences across models, tasks, and domains, with current LLMs showing capabilities in understanding ontological knowledge but weaknesses in reasoning and learning tasks. Further experiments with few-shot and chain-of-thought prompting illustrate how different prompting strategies affect model performance. Additionally, a human evaluation reveals that LLMs outperform humans in understanding and reasoning tasks but fall short in most learning tasks. These findings highlight both the potential and limitations of LLMs in processing symbolic knowledge and establish OntoURL as a critical benchmark for advancing the integration of LLMs with formal knowledge representations.

Figures (8)

• Figure 1: A sub-ontology excerpt from the Conference Ontology for representing academic conferences, illustrating the hierarchical structure of classes (in green) and instances (in blue). Most of the classes, relations, instances, and semantic information are omitted for the clarity.
• Figure 2: The taxonomy of LLM ontological capabilities, inspired by Bloom’s taxonomy. Each capability is positioned within a triangular structure and briefly explained on the right.
• Figure 3: The pipeline of OntoURL construction: (1) elements are extracted from ontologies using rule-based extraction (understanding and learning tasks) and an ontology reasoner HermiT (reasoning tasks); (2) the extracted elements are transformed into natural language questions; (3) distractors are added to form multiple-choice questions; and (4) the generated data is filtered and evaluated.
• Figure 4: Question distribution of OntoURL tasks and domains. Additional statistics, such as average lengths of questions, options, and answers, are provided in Appendix \ref{['app:statistics']}.
• Figure 5: The performance of Qwen2.5-7B, 72B (green and blue) and four domain-specific LLMs (yellow) across four domains.