Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Testing Database Systems with Large Language Model Synthesized Fragments

Suyang Zhong, Manuel Rigger

TL;DR

DBMS bug detection faces dialect gaps and high maintenance costs for dialect-aware generators. ShQveL introduces SQL sketches to integrate LLM-synthesized fragments into existing generators, enabling automatic learning of dialect-specific features and their validation through execution feedback. In a six-month, multi-DBMS evaluation, ShQveL discovered 55 unique bugs across 5 DBMSs, with 50 fixed by developers, and achieved competitive or superior code coverage compared to manual generators on several systems. The approach offers practical scalability, reuse of learned features, and an efficient learning phase that reduces LLM calls while maintaining robust bug detection capabilities.

Abstract

Various automated testing approaches have been proposed for Database Management Systems (DBMSs). Many such approaches generate pairs of equivalent queries to identify bugs that cause DBMSs to compute incorrect results, and have found hundreds of bugs in mature, widely used DBMSs. Most of these approaches are based on manually written SQL generators; however, their bug-finding capabilities remain constrained by the limited set of SQL features supported by the generators. In this work, we propose ShQveL, an approach that augments existing SQL test-case generators by leveraging Large Language Models (LLMs) to synthesize SQL fragments. Our key idea is to systematically incorporate SQL features gained through automated interactions with LLMs into the SQL generators, increasing the features covered while efficiently generating test cases. Specifically, ShQveL uses SQL sketches -- SQL statements with incomplete code segments that LLMs fill -- to integrate LLM-generated content into the generator. We evaluated ShQveL on 5 DBMSs and discovered 55 unique and previously unknown bugs, 50 of which were promptly fixed after our reports.

Testing Database Systems with Large Language Model Synthesized Fragments

TL;DR

DBMS bug detection faces dialect gaps and high maintenance costs for dialect-aware generators. ShQveL introduces SQL sketches to integrate LLM-synthesized fragments into existing generators, enabling automatic learning of dialect-specific features and their validation through execution feedback. In a six-month, multi-DBMS evaluation, ShQveL discovered 55 unique bugs across 5 DBMSs, with 50 fixed by developers, and achieved competitive or superior code coverage compared to manual generators on several systems. The approach offers practical scalability, reuse of learned features, and an efficient learning phase that reduces LLM calls while maintaining robust bug detection capabilities.

Abstract

Various automated testing approaches have been proposed for Database Management Systems (DBMSs). Many such approaches generate pairs of equivalent queries to identify bugs that cause DBMSs to compute incorrect results, and have found hundreds of bugs in mature, widely used DBMSs. Most of these approaches are based on manually written SQL generators; however, their bug-finding capabilities remain constrained by the limited set of SQL features supported by the generators. In this work, we propose ShQveL, an approach that augments existing SQL test-case generators by leveraging Large Language Models (LLMs) to synthesize SQL fragments. Our key idea is to systematically incorporate SQL features gained through automated interactions with LLMs into the SQL generators, increasing the features covered while efficiently generating test cases. Specifically, ShQveL uses SQL sketches -- SQL statements with incomplete code segments that LLMs fill -- to integrate LLM-generated content into the generator. We evaluated ShQveL on 5 DBMSs and discovered 55 unique and previously unknown bugs, 50 of which were promptly fixed after our reports.
Paper Structure (55 sections, 4 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 55 sections, 4 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. SQLancer++
  4. Motivation
  5. LLM-based Test Generation
  6. Methodology
  7. Results
  8. LLM-based Generator Generation
  9. Methodology
  10. Results
  11. ShQveL
  12. System overview
  13. Sketch Generation
  14. Sketch Design
  15. Context statements
  16. ...and 40 more sections

Figures (4)

  • Figure 1: Overview of ShQveL
  • Figure 2: Unique bugs found on CrateDB for one hour across 5 runs. The shadow shows the standard deviation.
  • Figure 3: The cost of LLM APIs to learn features on SQLite, DuckDB, and PostgreSQL over six hours.
  • Figure 4: The cumulative validity rate of test cases executing ShQveL on SQLite, DuckDB, and PostgreSQL over six hours.