BugForge: Constructing and Utilizing DBMS Bug Repository to Enhance DBMS Testing

Abstract

DBMSs are complex systems prone to bugs that may lead to system failures or compromise data integrity. Establishing unified DBMS bug repositories is crucial for systematically organizing bug-related data, enabling code improvement, and supporting automated testing. In particular, bug reports often contain valuable test inputs and bug-triggering clues that help explore rare execution paths and expose critical buggy behavior, thereby guiding automated DBMS testing. However, the heterogeneity of bug reports, along with their incomplete or inaccurate content, makes it challenging to build unified repositories and convert them into high-quality test cases. In this paper, we propose BugForge, a framework that constructs standardized DBMS bug repositories and leverages them to generate high-quality test cases to enhance DBMS testing. Specifically, BugForge progressively collects bug reports, then employs syntax-aware processing and input-adaptive raw PoC extraction to construct a DBMS bug repository. The repository stores structured bug-related data, including bug metadata and raw PoCs that entail potential bug-triggering semantics. These data are further refined into high-quality test cases through semantic-guided adaptation, thereby enabling enhanced DBMS testing methods, including DBMS fuzzing, regression testing, and cross-DBMS bug discovery. We implemented BugForge for PostgreSQL, MySQL, MariaDB, and MonetDB, totally integrated 37,632 bug reports spanning up to 28 years. Based on the repository, BugForge uncovered 35 previously unknown bugs with 22 confirmed by developers, demonstrating the value of constructing and utilizing bug repositories for DBMS testing.

Figures (10)

• Figure 1: MySQL Bug Report #110804 contains a raw PoC that is difficult to extract automatically. This report complicates automated analysis, as the PoC is conveyed in prose instead of being explicitly presented in a distinct code block. The sequence of statements to reproduce the bug, must be manually reconstructed from the content of bug report.
• Figure 2: MySQL Bug Report #102205 misses the configuration required for successful execution. The bug is unreproducible because the bug report lacks the configuration information for the log_bin_trust_function_creators system variable, but it triggers the expected result after BugForge adapts the raw PoC.
• Figure 3: The overview design of BugForge. BugForge is an end-to-end framework for DBMS bug repository construction and utilization. It progressively collects bug reports, applies syntax-aware processing and the input-adaptive LLM pipeline with RAG to extract raw PoCs and constructs a DBMS bug repository. Next, BugForge converts raw PoCs into high-quality test cases through semantic-guided adaptation under a stable environment and subsequently employs these cases in DBMS testing (e.g., fuzzing, regression, and cross-DBMS testing).
• Figure 4: An example prompt for LLM in raw PoC extraction. It consists of four components: (1) System Prompt, which instructs LLM to analyze bug reports. (2) Extraction Prompt, which guides the LLM through extraction, on-demand context expansion, and self-examination. (3) Reference Prompt, which retrieves positive and negative exemplars from the library. (4) Task Prompt, which supplies the bug summary, raw PoC-related fragments, and retrieved references for the target report.
• Figure 5: An example case in BugForge repository.BugForge normalizes heterogeneous DBMS bug reports into a structured representation. Each case includes: (1) basic metadata of the report, (2) summarized bug-type labels and the PoC-related fragments, (3) the extracted raw PoC used for downstream adaptation and DBMS testing.