DSTI at LLMs4OL 2024 Task A: Intrinsic versus extrinsic knowledge for type classification

TL;DR

This work compares intrinsic knowledge in large language models with extrinsic, externally grounded knowledge represented as semantic towers for ontology learning, evaluated on WordNet and GeoNames within the LLMs4OL 2024 challenge. Semantic towers are built from domain semantic primitives and embedded into a 1024-dimensional vector space to enable groundings via a retrieval-like process, while fine-tuned Flan-T5-small models provide a strong intrinsic baseline. Empirical results show that intrinsic models generally outperform extrinsic towers on official test splits, but the semantic towers improve grounding for certain nuanced classifications (e.g., adverbs, plural forms) and can enable finer-grained typing in some cases. The findings highlight a trade-off between raw performance and semantic grounding, suggesting future work to broaden the integration of semantic towers with LLM-based ontology learning systems for improved grounding and interpretability.

Abstract

We introduce semantic towers, an extrinsic knowledge representation method, and compare it to intrinsic knowledge in large language models for ontology learning. Our experiments show a trade-off between performance and semantic grounding for extrinsic knowledge compared to a fine-tuned model intrinsic knowledge. We report our findings on the Large Language Models for Ontology Learning (LLMs4OL) 2024 challenge.

Figures (4)

• Figure 1: WordNet and GeoNames semantic towers with examples.
• Figure 2: Subtask A.1 term typing WordNet examples.
• Figure 3: Subtask A.2 term typing GeoNames examples.
• Figure 4: RAG system architecture.