Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance

Felix Brei, Johannes Frey, Lars-Peter Meyer

TL;DR

This study investigates whether sub-1B parameter language models can be fine-tuned to translate natural language questions into SPARQL queries, enabling offline AI-assisted querying on commodity hardware. It evaluates three model families (BART, M2M100, NLLB-200) across three datasets (Organizational Graph, CoyPu subset, QALD10) and finds that BART and M2M100 variants can learn NL-to-SPARQL mappings to varying degrees, while T5 variants underperform on these tasks. Results reveal strong dataset dependence and highlight issues such as entity linking and prefix requirements, underscoring prerequisites for effective fine-tuning and data generation. The work suggests practical implications for resilient, on-device semantic web tools and outlines future directions including integration with retrieval-augmented generation, open-source data generation, and model optimization for on-device deployment.

Abstract

In this work we will show that language models with less than one billion parameters can be used to translate natural language to SPARQL queries after fine-tuning. Using three different datasets ranging from academic to real world, we identify prerequisites that the training data must fulfill in order for the training to be successful. The goal is to empower users of semantic web technology to use AI assistance with affordable commodity hardware, making them more resilient against external factors.

Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance

TL;DR

This study investigates whether sub-1B parameter language models can be fine-tuned to translate natural language questions into SPARQL queries, enabling offline AI-assisted querying on commodity hardware. It evaluates three model families (BART, M2M100, NLLB-200) across three datasets (Organizational Graph, CoyPu subset, QALD10) and finds that BART and M2M100 variants can learn NL-to-SPARQL mappings to varying degrees, while T5 variants underperform on these tasks. Results reveal strong dataset dependence and highlight issues such as entity linking and prefix requirements, underscoring prerequisites for effective fine-tuning and data generation. The work suggests practical implications for resilient, on-device semantic web tools and outlines future directions including integration with retrieval-augmented generation, open-source data generation, and model optimization for on-device deployment.

Abstract

In this work we will show that language models with less than one billion parameters can be used to translate natural language to SPARQL queries after fine-tuning. Using three different datasets ranging from academic to real world, we identify prerequisites that the training data must fulfill in order for the training to be successful. The goal is to empower users of semantic web technology to use AI assistance with affordable commodity hardware, making them more resilient against external factors.
Paper Structure (18 sections, 3 figures, 5 tables)

This paper contains 18 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experimental Setup
  4. Model families
  5. T5 and Flan-T5
  6. BART
  7. M2M100 and NLLB-200
  8. Datasets used for Fine Tuning and Evaluation
  9. Organizational Graph
  10. A subset of the CoyPu graph
  11. QALD10
  12. Fine-tuning
  13. Results
  14. Organizational Graph
  15. CoyPu
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Number of correct SPARQL translations across epochs for the organizational graph for one fine-tuning run per model. 16 questions were presented.
  • Figure 2: Number of correct SPARQL translations across epochs for the CoyPu dataset for one fine-tuning run per model. 26 questions were presented.
  • Figure 3: Repeating the experiment ten times for both datasets then selecting the best model from each run