From Prompt to Graph: Comparing LLM-Based Information Extraction Strategies in Domain-Specific Ontology Development
Xuan Liu, Ziyu Li, Mu He, Ziyang Ma, Xiaoxu Wu, Gizem Yilmaz, Yiyuan Xia, Bingbing Li, He Tan, Jerry Ying Hsi Fuh, Wen Feng Lu, Anders E. W. Jarfors, Per Jansson
TL;DR
This paper addresses efficient ontology construction in domain-specific manufacturing by evaluating three LLM-based information extraction strategies for term and relation extraction in the casting domain. The authors implement a semi-automated workflow: data distillation via Retrieval-Augmented Generation, three extraction methods (pre-trained LLM, ICL, fine-tuning), and encoding of extracted term–relation triples into a Neo4j ontology with Cypher. They provide a rigorous ground-truth dataset with thousands of labeled terms and relations, and compare performance to identify best-performing method for casting knowledge extraction. The resulting casting ontology is validated by domain experts, demonstrating practical viability for knowledge management and digitalization in manufacturing. The study contributes a side-by-side methodological comparison and a prompt-based relation extraction approach tailored for manufacturing ontologies.
Abstract
Ontologies are essential for structuring domain knowledge, improving accessibility, sharing, and reuse. However, traditional ontology construction relies on manual annotation and conventional natural language processing (NLP) techniques, making the process labour-intensive and costly, especially in specialised fields like casting manufacturing. The rise of Large Language Models (LLMs) offers new possibilities for automating knowledge extraction. This study investigates three LLM-based approaches, including pre-trained LLM-driven method, in-context learning (ICL) method and fine-tuning method to extract terms and relations from domain-specific texts using limited data. We compare their performances and use the best-performing method to build a casting ontology that validated by domian expert.