Q-NL Verifier: Leveraging Synthetic Data for Robust Knowledge Graph Question Answering

TL;DR

The paper tackles the data bottleneck in KGQA by introducing Q-NL Verifier, a framework that generates synthetic SPARQL→NL translations with LLMs and uses a learned semantic verifier to filter for correctness. The verifier, implemented as bi-encoder and cross-encoder architectures, scores translations with a normalized similarity $s\in[0,1]$, enabling automatic quality control without gold references. Empirical results on LC-QuAD 2.0 show strong generalization across LLMs and human translations, with the verifier outperforming traditional NLP metrics in ranking translations by semantic correctness (rank correlation $\rho(ACC_{manual},\cdot)=1.0$). The approach also improves NL→Q translation when used to train or filter data, and the authors release LC-QuAD 2.0-synth with verifier scores to support robust QA research. Overall, Q-NL Verifier enables scalable, semantically accurate QA data generation and provides practical benefits for dataset cleaning, model training, and live QA systems.

Abstract

Question answering (QA) requires accurately aligning user questions with structured queries, a process often limited by the scarcity of high-quality query-natural language (Q-NL) pairs. To overcome this, we present Q-NL Verifier, an approach to generating high-quality synthetic pairs of queries and NL translations. Our approach relies on large language models (LLMs) to generate semantically precise natural language paraphrases of structured queries. Building on these synthetic Q-NL pairs, we introduce a learned verifier component that automatically determines whether a generated paraphrase is semantically equivalent to the original query. Our experiments with the well-known LC-QuAD 2.0 benchmark show that Q-NL Verifier generalizes well to paraphrases from other models and even human-authored translations. Our approach strongly aligns with human judgments across varying query complexities and outperforms existing NLP metrics in assessing semantic correctness. We also integrate the verifier into QA pipelines, showing that verifier-filtered synthetic data has significantly higher quality in terms of translation correctness and enhances NL to Q translation accuracy. Lastly, we release an updated version of the LC-QuAD 2.0 benchmark containing our synthetic Q-NL pairs and verifier scores, offering a new resource for robust and scalable QA.

Figures (7)

• Figure 1: Motivating example. 1.) GPT-4o cannot translate an NL question into a query, and our verifier predicts a low correctness score. 2.) GPT-4o can directly translate a query into natural language, and our verifier predicts correctness with high certainty.
• Figure 2: Our approach: Q-NL Verifier. (1) Pipeline for synthetic SPARQL/NL pair generation by tackling the Q $\rightarrow$ NL task. (2) The pairs previously generated are used to train the verifier model.
• Figure 3: Sequence of prompts and LLM answers produced by our approach during Q $\rightarrow$ NL generation. The SPARQL query is taken from the LC-QuAD 2.0 benchmark.
• Figure 4: Verifier models for measuring the similarity between a query and its natural language translation. (1) A Bi-Encoder processes the query and translation independently into latent vectors, and their similarity is measured using cosine similarity. (2) A Cross-Encoder jointly processes the query and translation within one model and predicts similarity as a scalar value between 0 and 1 via a classifier.
• Figure 5: Examples of translations with different LLMs