Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems

Alexander Berndt, Thomas Bach, Rainer Gemulla, Marcus Kessel, Sebastian Baltes

TL;DR

The paper investigates the flakiness of LLM-generated tests in both industrial and open-source DBMS contexts, revealing that generated tests can be slightly more flaky than existing ones. Using a meta-generation approach with two LLMs, the study analyzes native C++ and SQL tests across SAP HANA, DuckDB, MySQL, and SQLite, showing unordered-collection–driven flakiness as the primary cause and demonstrating transfer of existing flaky behavior via prompt context. It also shows that closed-source systems like SAP HANA are more susceptible to flakiness transfer and that merely providing more context to LLMs is not a sufficient safeguard. The findings highlight the need for tailored, system-specific context and robust validation when employing LLMs for test generation, to mitigate non-functional flaws and maintain CI reliability.

Abstract

Flaky tests are a common problem in software testing. They produce inconsistent results when executed multiple times on the same code, invalidating the assumption that a test failure indicates a software defect. Recent work on LLM-based test generation has identified flakiness as a potential problem with generated tests. However, its prevalence and underlying causes are unclear. We examined the flakiness of LLM-generated tests in the context of four relational database management systems: SAP HANA, DuckDB, MySQL, and SQLite. We amplified test suites with two LLMs, GPT-4o and Mistral-Large-Instruct-2407, to assess the flakiness of the generated test cases. Our results suggest that generated tests have a slightly higher proportion of flaky tests compared to existing tests. Based on a manual inspection, we found that the most common root cause of flakiness was the reliance of a test on a certain order that is not guaranteed ("unordered collection"), which was present in 72 of 115 flaky tests (63%). Furthermore, both LLMs transferred the flakiness from the existing tests to the newly generated tests via the provided prompt context. Our experiments suggest that flakiness transfer is more prevalent in closed-source systems such as SAP HANA than in open-source systems. Our study informs developers on what types of flakiness to expect from LLM-generated tests. It also highlights the importance of providing LLMs with tailored context when employing LLMs for test generation.

On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems

TL;DR

The paper investigates the flakiness of LLM-generated tests in both industrial and open-source DBMS contexts, revealing that generated tests can be slightly more flaky than existing ones. Using a meta-generation approach with two LLMs, the study analyzes native C++ and SQL tests across SAP HANA, DuckDB, MySQL, and SQLite, showing unordered-collection–driven flakiness as the primary cause and demonstrating transfer of existing flaky behavior via prompt context. It also shows that closed-source systems like SAP HANA are more susceptible to flakiness transfer and that merely providing more context to LLMs is not a sufficient safeguard. The findings highlight the need for tailored, system-specific context and robust validation when employing LLMs for test generation, to mitigate non-functional flaws and maintain CI reliability.

Abstract

Flaky tests are a common problem in software testing. They produce inconsistent results when executed multiple times on the same code, invalidating the assumption that a test failure indicates a software defect. Recent work on LLM-based test generation has identified flakiness as a potential problem with generated tests. However, its prevalence and underlying causes are unclear. We examined the flakiness of LLM-generated tests in the context of four relational database management systems: SAP HANA, DuckDB, MySQL, and SQLite. We amplified test suites with two LLMs, GPT-4o and Mistral-Large-Instruct-2407, to assess the flakiness of the generated test cases. Our results suggest that generated tests have a slightly higher proportion of flaky tests compared to existing tests. Based on a manual inspection, we found that the most common root cause of flakiness was the reliance of a test on a certain order that is not guaranteed ("unordered collection"), which was present in 72 of 115 flaky tests (63%). Furthermore, both LLMs transferred the flakiness from the existing tests to the newly generated tests via the provided prompt context. Our experiments suggest that flakiness transfer is more prevalent in closed-source systems such as SAP HANA than in open-source systems. Our study informs developers on what types of flakiness to expect from LLM-generated tests. It also highlights the importance of providing LLMs with tailored context when employing LLMs for test generation.
Paper Structure (21 sections, 3 figures, 5 tables)

This paper contains 21 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Flakiness Definition
  4. Types of Flakiness
  5. Flakiness of LLM-Generated Tests
  6. Database Management Systems
  7. SAP HANA
  8. MySQL Server
  9. SQLite
  10. DuckDB
  11. Methodology and Research Questions
  12. RQ1: Prevalence
  13. RQ2: Root Causes
  14. RQ3: Transfer
  15. Threats to Validity
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Study setup inspired by meta-generation: We first prompt the LLM to generate new tests given an existing test file. Then, the generated code is added to the existing test file before compilation. After compilation, we identify the newly added, generated tests, execute them in separate Docker containers, and store the results in a database.
  • Figure 2: Violin plots displaying the syntactical difference between generated and existing tests.
  • Figure 3: Venn diagram showing the number of files containing generated flaky tests for GPT-4o and Mistral, respectively.