SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Task

TL;DR

This paper tackles the Text2Cypher problem by proposing SyntheT2C, a dual-pipeline framework that generates high-quality synthetic Question-Cypher pairs to fine-tune LLMs for Cypher query generation. It combines an LLM-based prompting pipeline with a template-filling pipeline to produce diverse and syntactically rich data, followed by a robust automatic and manual validation workflow. The authors validate on two medical knowledge graphs, LHY and Hetionet, resulting in MedT2C (3000 training samples) that improves Cypher writing and execution accuracy across multiple LLMs, with scaling and ablation studies supporting robustness and the complementary value of both pipelines and validators. MedT2C and the SyntheT2C codebase are open-sourced, highlighting the method’s practicality and potential applicability to other Neo4j databases and structured query tasks.

Abstract

Integrating Large Language Models (LLMs) with existing Knowledge Graph (KG) databases presents a promising avenue for enhancing LLMs' efficacy and mitigating their "hallucinations". Given that most KGs reside in graph databases accessible solely through specialized query languages (e.g., Cypher), it is critical to connect LLMs with KG databases by automating the translation of natural language into Cypher queries (termed as "Text2Cypher" task). Prior efforts tried to bolster LLMs' proficiency in Cypher generation through Supervised Fine-Tuning (SFT). However, these explorations are hindered by the lack of annotated datasets of Query-Cypher pairs, resulting from the labor-intensive and domain-specific nature of such annotation. In this study, we propose SyntheT2C, a methodology for constructing a synthetic Query-Cypher pair dataset, comprising two distinct pipelines: (1) LLM-based prompting and (2) template-filling. SyntheT2C is applied to two medical KG databases, culminating in the creation of a synthetic dataset, MedT2C. Comprehensive experiments demonstrate that the MedT2C dataset effectively enhances the performance of backbone LLMs on Text2Cypher task via SFT. Both the SyntheT2C codebase and the MedT2C dataset are released in https://github.com/ZGChung/SyntheT2C.

Figures (15)

• Figure 1: SyntheT2C builds synthetic data with two pipelines to SFT LLMs so that their performance on Text2Cypher task is enhanced.
• Figure 2: Workflow of two pipelines inside SyntheT2C.
• Figure 3: Illustration of the automatic validators.
• Figure 4: Example template in Template-filling pipeline.
• Figure 5: Result of Supervised Fine-Tuning each LLMs with MedT2C. Accuracy annotations marked in white box.