CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Setting

TL;DR

CompoST investigates whether large language models map compositional questions to SPARQL queries in a systematic way within the QALD setting. The authors operationalize compositionality into systematicity, generate CompoST, three graph-pattern datasets with depth/breadth variations, and verbalize them via Lemon lexica to test LLMs under zero-shot, few-shot, and fine-tuning regimes. Across over 400 experiments, results show macro $F_1$ scores degrade as pattern complexity increases, and compositionality-focused metrics remain low, even with self-contained inputs; fine-tuning can boost overall performance on simple cases but does not achieve robust systematic composition. The work highlights fundamental limitations in current LLMs' ability to interpret questions compositionally in knowledge-base QA and motivates future research into targeted training or hybrid approaches.

Abstract

Language interpretation is a compositional process, in which the meaning of more complex linguistic structures is inferred from the meaning of their parts. Large language models possess remarkable language interpretation capabilities and have been successfully applied to interpret questions by mapping them to SPARQL queries. An open question is how systematic this interpretation process is. Toward this question, in this paper, we propose a benchmark for investigating to what extent the abilities of LLMs to interpret questions are actually compositional. For this, we generate three datasets of varying difficulty based on graph patterns in DBpedia, relying on Lemon lexica for verbalization. Our datasets are created in a very controlled fashion in order to test the ability of LLMs to interpret structurally complex questions, given that they have seen the atomic building blocks. This allows us to evaluate to what degree LLMs are able to interpret complex questions for which they "understand" the atomic parts. We conduct experiments with models of different sizes using both various prompt and few-shot optimization techniques as well as fine-tuning. Our results show that performance in terms of macro $F_1$ degrades from $0.45$ over $0.26$ down to $0.09$ with increasing deviation from the samples optimized on. Even when all necessary information was provided to the model in the input, the $F_1$ scores do not exceed $0.57$ for the dataset of lowest complexity. We thus conclude that LLMs struggle to systematically and compositionally interpret questions and map them into SPARQL queries.

Figures (5)

• Figure 1: RDF graph of SPARQL graph pattern instance with depth of $3$ and breadth of $3$ with concrete entities and properties from DBpedia.
• Figure 2: Example question-query-pairs. $1$ and $2$ together contain all relevant information to compose the bottom query. Only triple patterns of queries shown.
• Figure 3: EBNF bnf of the sentences generated for the CompoST dataset.
• Figure 4: Macro $F_1$ scores of best approaches of the respective category for the hard dataset, grouped by graph pattern depth and breadth. Training data comprised samples up to two edges, i.e., up to breadth $2$, depth $1$ and depth $2$, breadth $1$.
• Figure 5: Results for a depth and breadth $3$ graph pattern instance and connected sub-patterns taken from the best fine-tuned medium approach. Arrows indicate source contains target node pattern, omitting nodes with $>1$ edge difference.

Theorems & Definitions (2)

• definition thmcounterdefinition: Productivity comp
• definition thmcounterdefinition: Systematicity comp