Improving LLM-based Ontology Matching with fine-tuning on synthetic data

TL;DR

The paper tackles the challenge of complex ontology matching using LLMs by proposing an instruction-fine-tuning workflow trained on synthetically generated data. It introduces a space-reduction pipeline to extract subontologies, prompts the LLM to produce partial EDOAL alignments, and aggregates results into a final alignment, enabling scalable and structured outputs. The findings show that synthetic-data fine-tuning improves performance on simple 1:1 mappings and can boost precision for complex alignments via cross-validation, though generalization to nmappings remains limited. The work highlights the promise of synthetic data for adapting LLMs to ontology matching and outlines directions for improving data quality, multilingual capabilities, and integration of reasoning components.

Abstract

Large Language Models (LLMs) are increasingly being integrated into various components of Ontology Matching pipelines. This paper investigates the capability of LLMs to perform ontology matching directly on ontology modules and generate the corresponding alignments. Furthermore, it is explored how a dedicated fine-tuning strategy can enhance the model's matching performance in a zero-shot setting. The proposed method incorporates a search space reduction technique to select relevant subsets from both source and target ontologies, which are then used to automatically construct prompts. Recognizing the scarcity of reference alignments for training, a novel LLM-based approach is introduced for generating a synthetic dataset. This process creates a corpus of ontology submodule pairs and their corresponding reference alignments, specifically designed to fine-tune an LLM for the ontology matching task. The proposed approach was evaluated on the Conference, Geolink, Enslaved, Taxon, and Hydrography datasets from the OAEI complex track. The results demonstrate that the LLM fine-tuned on the synthetically generated data exhibits superior performance compared to the non-fine-tuned base model. The key contribution is a strategy that combines automatic dataset generation with fine-tuning to effectively adapt LLMs for ontology matching tasks.

Figures (5)

• Figure 1: Overview of the proposed LLM-based ontology alignment pipeline. Starting from a source and target ontology, queries are automatically generated from the source ontology using a PageRank-based method. These queries are then used to select subontology parts to construct multiple prompts for direct LLM alignment, which generate multiple EDOAL alignments. These alignments are then post-processed through a repair and merging step to produce the final EDOAL alignment.
• Figure 2: Alignment and ontology generation workflow. Starting from a seed alignment template, the LLM is prompted to fill in the masked placeholders with entities while preserving semantic equivalence. Once the alignment is completed, the LLM is then prompted to sequentially generate the ontologies that correspond to the proposed alignment.
• Figure 3: Distribution of prompt tokens .
• Figure 4: Performance of the models compared with the smaller but fine-tuned model.
• Figure 5: Average performance of the proposed approaches across all evaluated datasets. The table shows precision (p), recall (r), and F-measure (f) for both simple (s-p, s-r, s-f) and complex (c-p, c-r, c-f) correspondences.