Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search

Yujia Chen, Yingli Zhou, Fangyuan Zhang, Cuiyun Gao

Abstract

Database Management Systems (DBMSs) are fundamental infrastructure for modern data-driven applications, where thorough testing with high-quality SQL test cases is essential for ensuring system reliability. Traditional approaches such as fuzzing can be effective for specific DBMSs, but adapting them to different proprietary dialects requires substantial manual effort. Large Language Models (LLMs) present promising opportunities for automated SQL test generation, but face critical challenges in industrial environments. First, lightweight models are widely used in organizations due to security and privacy constraints, but they struggle to generate syntactically valid queries for proprietary SQL dialects. Second, LLM-generated queries are often semantically similar and exercise only shallow execution paths, thereby quickly reaching a coverage plateau. To address these challenges, we propose MIST, an LLM-based test case generatIon framework for DBMS through Monte Carlo Tree search. MIST consists of two stages: Feature-Guided Error-Driven Test Case Synthetization, which constructs a hierarchical feature tree and uses error feedback to guide LLM generation, aiming to produce syntactically valid and semantically diverse queries for different DBMS dialects, and Monte Carlo Tree Search-Based Test Case Mutation, which jointly optimizes seed query selection and mutation rule application guided by coverage feedback, aiming at boosting code coverage by exploring deeper execution paths. Experiments on three widely-used DBMSs with four lightweight LLMs show that MIST achieves average improvements of 43.3% in line coverage, 32.3% in function coverage, and 46.4% in branch coverage compared to the baseline approach with the highest line coverage of 69.3% in the Optimizer module.

LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search

Abstract

Database Management Systems (DBMSs) are fundamental infrastructure for modern data-driven applications, where thorough testing with high-quality SQL test cases is essential for ensuring system reliability. Traditional approaches such as fuzzing can be effective for specific DBMSs, but adapting them to different proprietary dialects requires substantial manual effort. Large Language Models (LLMs) present promising opportunities for automated SQL test generation, but face critical challenges in industrial environments. First, lightweight models are widely used in organizations due to security and privacy constraints, but they struggle to generate syntactically valid queries for proprietary SQL dialects. Second, LLM-generated queries are often semantically similar and exercise only shallow execution paths, thereby quickly reaching a coverage plateau. To address these challenges, we propose MIST, an LLM-based test case generatIon framework for DBMS through Monte Carlo Tree search. MIST consists of two stages: Feature-Guided Error-Driven Test Case Synthetization, which constructs a hierarchical feature tree and uses error feedback to guide LLM generation, aiming to produce syntactically valid and semantically diverse queries for different DBMS dialects, and Monte Carlo Tree Search-Based Test Case Mutation, which jointly optimizes seed query selection and mutation rule application guided by coverage feedback, aiming at boosting code coverage by exploring deeper execution paths. Experiments on three widely-used DBMSs with four lightweight LLMs show that MIST achieves average improvements of 43.3% in line coverage, 32.3% in function coverage, and 46.4% in branch coverage compared to the baseline approach with the highest line coverage of 69.3% in the Optimizer module.
Paper Structure (32 sections, 2 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 32 sections, 2 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. DBMS Testing
  4. Monte Carlo Tree Search
  5. Approach
  6. Overview
  7. Feature-Guided Error-Driven Test Case Synthetization
  8. Feature tree construction
  9. Hierarchical feature selection
  10. Error-Driven test case generation
  11. Monte Carlo Tree Search-Based Test Case Mutation
  12. Mutation rules curation
  13. MCTS-based mutation process
  14. Experimental Setup
  15. Research Questions
  16. ...and 17 more sections

Figures (6)

  • Figure 1: An illustrative example of using Qwen2.5-7B to generate a SQL test case for DBMS.
  • Figure 2: Illustrating challenges in LLM-based DBMS test case generation with Qwen2.5-7B.
  • Figure 3: The overview of MIST.
  • Figure 4: The prompt template for test case synthetization. [DBMS_TYPE] is the target DBMS, [FEATURE_LIST] contains selected features from the hierarchical tree, and [ERROR_MEMORY] provides feedback from previous execution failures.
  • Figure 5: Illustrating the MCTS for test case mutation.
  • ...and 1 more figures