ARUQULA -- An LLM based Text2SPARQL Approach using ReAct and Knowledge Graph Exploration Utilities

TL;DR

ARUQULA tackles Text2SPARQL for RDF knowledge graphs by extending SPINACH with ReAct-based reasoning and dedicated KG exploration utilities. A dual grounding strategy separates schema-term grounding from named-entity grounding, employing a hybrid vector search for concepts and a full-text Lucene index for entities. Evaluated on the TEXT2SPARQL challenge with CK25 and DB25, the work analyzes agent behavior, timing, and failure modes to reveal strengths and limitations of automatic evaluation in KGQA. The approach advances multilingual, multi-KG Text2SPARQL methods and points to practical directions for reducing latency, improving grounding reliability, and enabling cross-KG deployment.

Abstract

Interacting with knowledge graphs can be a daunting task for people without a background in computer science since the query language that is used (SPARQL) has a high barrier of entry. Large language models (LLMs) can lower that barrier by providing support in the form of Text2SPARQL translation. In this paper we introduce a generalized method based on SPINACH, an LLM backed agent that translates natural language questions to SPARQL queries not in a single shot, but as an iterative process of exploration and execution. We describe the overall architecture and reasoning behind our design decisions, and also conduct a thorough analysis of the agent behavior to gain insights into future areas for targeted improvements. This work was motivated by the Text2SPARQL challenge, a challenge that was held to facilitate improvements in the Text2SPARQL domain.

Figures (4)

• Figure 1: Architecture and Action Graph (in the blue area). At the controller step the LLM can decide whether to search, inspect, execute or stop.
• Figure 2: At each individual step of the agent we noted which action was chosen how often from the about 250 queries of the benchmark. The step index is shown on the x axis and the height of the bar corresponds to the number of times the agent has reached that step. The colors indicate the action selected at a step.
• Figure 3: A large portion of actions that the agent takes starting at step three is execute_sparql. This pie chart shows the ratios of actions that come directly after such an action, i.e. in three out of five times it is followed by yet another execute_sparql action and in one out of four times the agent decides to stop afterwards, whereas in one out of seven cases the agent resorts to retrieving information from the knowledge graph directly via search or get functions to refine future queries
• Figure 4: For this plot we grouped the actions into categories to increase readability. It shows the total number of times an action from each category was taken up until a certain step index (i.e. cumulative). For example, we can see that at step seven a total of almost 400 SPARQL queries was executed across all 250 questions. Most notably, only about 200 times total was the stop action called, meaning that circa 50 times the resulting SPARQL query was simply scraped from the conversation trace up until this point without confidence in its correctness.