Effective Bug Detection in Graph Database Engines: An LLM-based Approach
Jiayi Wu, Zhengyu Wu, Ronghua Li, Hongchao Qin, Guoren Wang
TL;DR
This paper tackles bug detection in graph database engines across multiple query languages by introducing DGDB, an LLM-driven paradigm that automatically generates diverse queries and applies differential testing across engines. DGDB comprises graph data generation, language-agnostic query generation via ChatGPT, and cross-engine bug detection, enabling applicability to Cypher and Gremlin. Empirical results show DGDB achieves a high non-empty-query ratio and uncovers multiple wrong-result bugs across Neo4j, AgensGraph, JanusGraph, and TinkerGraph, outperforming baselines in key metrics. The work demonstrates the practical value of LLM-guided query synthesis for robust cross-language bug detection in graph databases, with broad applicability to future engine versions and languages.
Abstract
Graph database engines play a pivotal role in efficiently storing and managing graph data across various domains, including bioinformatics, knowledge graphs, and recommender systems. Ensuring data accuracy within graph database engines is paramount, as inaccuracies can yield unreliable analytical outcomes. Current bug-detection approaches are confined to specific graph query languages, limiting their applicabilities when handling graph database engines that use various graph query languages across various domains. Moreover, they require extensive prior knowledge to generate queries for detecting bugs. To address these challenges, we introduces DGDB, a novel paradigm harnessing large language models(LLM), such as ChatGPT, for comprehensive bug detection in graph database engines. DGDB leverages ChatGPT to generate high-quality queries for different graph query languages. It subsequently employs differential testing to identify bugs in graph database engines. We applied this paradigm to graph database engines using the Gremlin query language and those using the Cypher query language, generating approximately 4,000 queries each. In the latest versions of Neo4j, Agensgraph, and JanusGraph databases, we detected 2, 5, and 3 wrong-result bugs, respectively.