Large Language Models as Oracles for Ontology Alignment

TL;DR

This work investigates using Large Language Models as selective Oracles to validate uncertain mappings in ontology alignment, integrating an LLM-based validator into the LogMap pipeline. By designing ontology-driven prompts that leverage lexical, contextual, and synonym information, and by restricting LLM usage to a subset of mappings $\mathcal{M}_{ask}$, the approach achieves competitive end-to-end performance on nine OAEI tasks, including a top-2 finish in the bio-ml track. The study provides a comprehensive evaluation across multiple LLMs and prompt templates, highlighting the diagnostic power of LLM-based Oracles (notably Gemini 2.5 Flash with $P_{ ext{S}}^{ ext{NLF}}$) and analyzing the effects of prompt design, model choice, and open-weight options. The results demonstrate feasibility and practical benefits for cost-aware, AI-assisted ontology alignment, while outlining limitations and directions for future work such as ensemble methods, retrieval-augmented prompting, few-shot prompts, and broader domain evaluation.

Abstract

There are many methods and systems to tackle the ontology alignment problem, yet a major challenge persists in producing high-quality mappings among a set of input ontologies. Adopting a human-in-the-loop approach during the alignment process has become essential in applications requiring very accurate mappings. However, user involvement is expensive when dealing with large ontologies. In this paper, we analyse the feasibility of using Large Language Models (LLM) to aid the ontology alignment problem. LLMs are used only in the validation of a subset of correspondences for which there is high uncertainty. We have conducted an extensive analysis over several tasks of the Ontology Alignment Evaluation Initiative (OAEI), reporting in this paper the performance of several state-of-the-art LLMs using different prompt templates. Using LLMs as Oracles resulted in strong performance in the OAEI 2025, achieving the top-2 overall rank in the bio-ml track.

Figures (8)

• Figure 1: LLM-in-the-loop as an Oracle to diagnose challenging matches in ontology alignment.
• Figure 2: Summary of the diagnostic results (Youden’s index) for the LLM-based Oracles.
• Figure 3: Comparison of LogMap, LogMap with Or$^{LLM}_{GF2.5}$, and LogMap in combination with Oracles with different error rates (Or$^{0}$, Or$^{20}$, and Or$^{30}$).
• Figure 4: Workflow of the ontology alignment system LogMap with calls to an Oracle.
• Figure 5: Diagnostic results (Youden’s index) by the LLM-based Oracles over the selected ontology matching tasks. For example, Flash 2.5-$\mathbf{P}\xspace_{\mathrm{EC+S}}^{\mathrm{NLF}}$ represents the LLM-based Oracle relying on the Gemini Flash 2.5 model and evaluated with the natural-language friendly (NLF) prompts with extended context (EC) and synonyms (S). We only completed a subset of experiments with GPT-4o Mini as a reference.