End-to-End Ontology Learning with Large Language Models

TL;DR

This work introduces OLLM, a general and scalable method for building the taxonomic backbone of an ontology from scratch, and introduces a novel suite of metrics for evaluating the quality of the generated ontology by measuring its semantic and structural similarity to the ground truth.

Abstract

Ontologies are useful for automatic machine processing of domain knowledge as they represent it in a structured format. Yet, constructing ontologies requires substantial manual effort. To automate part of this process, large language models (LLMs) have been applied to solve various subtasks of ontology learning. However, this partial ontology learning does not capture the interactions between subtasks. We address this gap by introducing OLLM, a general and scalable method for building the taxonomic backbone of an ontology from scratch. Rather than focusing on subtasks, like individual relations between entities, we model entire subcomponents of the target ontology by finetuning an LLM with a custom regulariser that reduces overfitting on high-frequency concepts. We introduce a novel suite of metrics for evaluating the quality of the generated ontology by measuring its semantic and structural similarity to the ground truth. In contrast to standard metrics, our metrics use deep learning techniques to define more robust distance measures between graphs. Both our quantitative and qualitative results on Wikipedia show that OLLM outperforms subtask composition methods, producing more semantically accurate ontologies while maintaining structural integrity. We further demonstrate that our model can be effectively adapted to new domains, like arXiv, needing only a small number of training examples. Our source code and datasets are available at https://github.com/andylolu2/ollm.

Figures (27)

• Figure 1: OLLM: Using annotations of documents with their relevant concepts, we train an LLM to model relevant subgraphs of the target ontology with a custom regulariser. During inference, the generated subgraphs for each document are summed and pruned to give the final output ontology. For evaluation, we measure the similarity between the generated ontology and the ground truth.
• Figure 2: Example subgraph induced for the Wikipedia page "Hybridity" (left), where $N = 4$ and $C = \{\text{Politics and race}, \text{Sociology of culture}\}$. The corresponding training text sequence (right), where text coloured in grey is ignored as training targets, but is still present as context for later tokens.
• Figure 3: Intersection of concepts among the train, validation and test splits of the datasets.
• Figure 4: Per token loss on a test set example of the final model trained with and without the custom masked loss objective. A stronger red colour represents a higher cross-entropy loss. Within the top-level concepts (children of the root) shown here, "Culture" and "Humanities" are in the training set while others are not. Using the masked loss objective improves generalisation on the high-level relations (e.g., "Main topic classifications" $\rightarrow$ "Academic disciplines") while maintaining performance on lower-level relations.
• Figure 5: Linearisation template for OLLM training.