LLMs4OL 2024 Overview: The 1st Large Language Models for Ontology Learning Challenge

TL;DR

This paper reports the first LLMs4OL Challenge at ISWC 2024, examining how large language models can support Ontology Learning. It defines three end-to-end tasks (Term Typing, Taxonomy Discovery, Non-Taxonomic Relation Extraction) and two evaluation phases (Few-shot, Zero-shot) with a public dataset and standard metrics including $P$, $R$, and $F1$. Results show diverse strategies—fine-tuning, prompt-tuning, and retrieval-augmented generation—across models such as GPT-3.5/4, LLaMA, BLOOM, BioMistral, and Mistral in domains like WordNet, GeoNames, UMLS, and GO. The study highlights the potential and remaining challenges of LLM-based OL and points to hybrid approaches and future work to scale, interpretability, and semantic-web impact.

Abstract

This paper outlines the LLMs4OL 2024, the first edition of the Large Language Models for Ontology Learning Challenge. LLMs4OL is a community development initiative collocated with the 23rd International Semantic Web Conference (ISWC) to explore the potential of Large Language Models (LLMs) in Ontology Learning (OL), a vital process for enhancing the web with structured knowledge to improve interoperability. By leveraging LLMs, the challenge aims to advance understanding and innovation in OL, aligning with the goals of the Semantic Web to create a more intelligent and user-friendly web. In this paper, we give an overview of the 2024 edition of the LLMs4OL challenge and summarize the contributions.

Figures (1)

• Figure 1: The LLMs4OL task paradigm is an end-to-end framework for ontology learning. The three OL tasks that empirically validated in the LLMs4OL 2024 challenge, based on our prior research babaei2023llms4ol, are depicted within the blue arrow, aligned with the greater LLMs4OL paradigm.