How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance

TL;DR

This work probes how LLM size influences Knowledge Graph Engineering tasks by reanalyzing a large, KG-focused benchmark (LLM-KG-Bench) across 26 open LLMs and 23 RDF/SPARQL task variations. It shows that scaling laws generally apply to KGE tasks but with notable plateaus and internal drops within model families, suggesting cost-aware model selection—sometimes mid-sized or code-specialized models offer best value. The findings highlight that open LLMs struggle on certain output-heavy KG tasks (RdfFriendCount, Text2Sparql), while code-specialized and proprietary models can bridge some gaps; MoE models do not guarantee superior performance. The study provides practical guidance for practitioners to balance performance and resource costs and outlines directions for extending LLM-KG-Bench to further dissect scaling effects and architectural influences.

Abstract

When using Large Language Models (LLMs) to support Knowledge Graph Engineering (KGE), one of the first indications when searching for an appropriate model is its size. According to the scaling laws, larger models typically show higher capabilities. However, in practice, resource costs are also an important factor and thus it makes sense to consider the ratio between model performance and costs. The LLM-KG-Bench framework enables the comparison of LLMs in the context of KGE tasks and assesses their capabilities of understanding and producing KGs and KG queries. Based on a dataset created in an LLM-KG-Bench run covering 26 open state-of-the-art LLMs, we explore the model size scaling laws specific to KGE tasks. In our analyses, we assess how benchmark scores evolve between different model size categories. Additionally, we inspect how the general score development of single models and families of models correlates to their size. Our analyses revealed that, with a few exceptions, the model size scaling laws generally also apply to the selected KGE tasks. However, in some cases, plateau or ceiling effects occurred, i.e., the task performance did not change much between a model and the next larger model. In these cases, smaller models could be considered to achieve high cost-effectiveness. Regarding models of the same family, sometimes larger models performed worse than smaller models of the same family. These effects occurred only locally. Hence it is advisable to additionally test the next smallest and largest model of the same family.

Figures (1)

• Figure 1: Plots showing the achieved central measure mean scores per model in relation to the model size for all tasks. Models of the same family are connected through a dashed line. As a reference, for each task, aösp the highest score achieved by a proprietary model included in the benchmark run is given and depicted in a red horizontal line.