Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parser Knows Best: Testing DBMS with Coverage-Guided Grammar-Rule Traversal

Yu Liang, Hong Hu

TL;DR

This paper introduces ParserFuzz, a generation-based fuzzing framework for DBMS testing that learns SQL grammar rules directly from DBMS parser definitions and uses code coverage as feedback to guide grammar-rule mutations. By exhaustively traversing grammar rules across multiple DBMSs without seed corpora, ParserFuzz achieves broad syntax feature coverage and uncovers memory-corruption and assertion bugs that outpace state-of-the-art tools. The evaluation across five popular DBMSs reports 81 previously unknown bugs, with 34 confirmed fixes, and demonstrates superior grammar and code coverage relative to baselines like Squirrel, SQLsmith, and SQLancer+QPG. The work highlights the practical impact of parser-rule-based query generation combined with coverage feedback for robust DBMS testing and provides open-source access for adoption and further research.

Abstract

Database Management System (DBMS) is the key component for data-intensive applications. Recently, researchers propose many tools to comprehensively test DBMS systems for finding various bugs. However, these tools only cover a small subset of diverse syntax elements defined in DBMS-specific SQL dialects, leaving a large number of features unexplored. In this paper, we propose ParserFuzz, a novel fuzzing framework that automatically extracts grammar rules from DBMSs' built-in syntax definition files for SQL query generation. Without any input corpus, ParserFuzz can generate diverse query statements to saturate the grammar features of the tested DBMSs, which grammar features could be missed by previous tools. Additionally, ParserFuzz utilizes code coverage as feedback to guide the query mutation, which combines different DBMS features extracted from the syntax rules to find more function and safety bugs. In our evaluation, ParserFuzz outperforms all state-of-the-art existing DBMS testing tools in terms of bug finding, grammar rule coverage and code coverage. ParserFuzz detects 81 previously unknown bugs in total across 5 popular DBMSs, where all bugs are confirmed and 34 have been fixed.

Parser Knows Best: Testing DBMS with Coverage-Guided Grammar-Rule Traversal

TL;DR

This paper introduces ParserFuzz, a generation-based fuzzing framework for DBMS testing that learns SQL grammar rules directly from DBMS parser definitions and uses code coverage as feedback to guide grammar-rule mutations. By exhaustively traversing grammar rules across multiple DBMSs without seed corpora, ParserFuzz achieves broad syntax feature coverage and uncovers memory-corruption and assertion bugs that outpace state-of-the-art tools. The evaluation across five popular DBMSs reports 81 previously unknown bugs, with 34 confirmed fixes, and demonstrates superior grammar and code coverage relative to baselines like Squirrel, SQLsmith, and SQLancer+QPG. The work highlights the practical impact of parser-rule-based query generation combined with coverage feedback for robust DBMS testing and provides open-source access for adoption and further research.

Abstract

Database Management System (DBMS) is the key component for data-intensive applications. Recently, researchers propose many tools to comprehensively test DBMS systems for finding various bugs. However, these tools only cover a small subset of diverse syntax elements defined in DBMS-specific SQL dialects, leaving a large number of features unexplored. In this paper, we propose ParserFuzz, a novel fuzzing framework that automatically extracts grammar rules from DBMSs' built-in syntax definition files for SQL query generation. Without any input corpus, ParserFuzz can generate diverse query statements to saturate the grammar features of the tested DBMSs, which grammar features could be missed by previous tools. Additionally, ParserFuzz utilizes code coverage as feedback to guide the query mutation, which combines different DBMS features extracted from the syntax rules to find more function and safety bugs. In our evaluation, ParserFuzz outperforms all state-of-the-art existing DBMS testing tools in terms of bug finding, grammar rule coverage and code coverage. ParserFuzz detects 81 previously unknown bugs in total across 5 popular DBMSs, where all bugs are confirmed and 34 have been fixed.

Paper Structure

This paper contains 18 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background & Challenges
  3. An Example Memory Error from MySQL
  4. Generation-based DBMS Testing Tools
  5. Mutation-based DBMS Testing Tools
  6. Parser Generators in the DBMSs
  7. Design of [0.5]ParserFuzz
  8. Parser Rule-based Query Generation
  9. Path Explosion due to Recursive Keyword
  10. Query Mutation with Coverage Feedback
  11. Implementation
  12. Evaluation
  13. DBMS Bugs
  14. Comparison with Existing Tools
  15. Contribution of Coverage Feedback
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of [0.5]ParserFuzz. It automatically extracts SQL features from DBMS-specific syntax definition files, and utilizes code coverage to select promising rules for query generation.
  • Figure 2: Evaluation of different testing tools on [0.5]MySQL, [0.5]CockroachDB, [0.5]TiDB, [0.5]MariaDB and [0.5]SQLite.