Conformance Testing of Relational DBMS Against SQL Specifications
Shuang Liu, Chenglin Tian, Jun Sun, Ruifeng Wang, Wei Lu, Yongxin Zhao, Yinxing Xue, Junjie Wang, Xiaoyong Du
TL;DR
This work tackles the challenge of ensuring RDBMS behavior conforms to the SQL specification by introducing SemConT, a semantic conformance testing framework that encodes SQL as executable Prolog semantics to serve as an oracle. It introduces three coverage criteria—keyword, rule, and composite rule coverage—to guide test-query generation and uses SQLancer seeds with targeted mutations to achieve broad semantic coverage. Evaluated on six widely used DBMS, SemConT uncovers $19$ bugs (with $18$ new) and $13$ inconsistencies, including many arising from under-specified or unclear SQL semantics, outperforming state-of-the-art metamorphic approaches. The results demonstrate the utility of executable SQL semantics for rigorous conformance testing and motivate improvements in both DBMS implementations and the SQL standard.
Abstract
A Relational Database Management System (RDBMS) is one of the fundamental software that supports a wide range of applications, making it critical to identify bugs within these systems. There has been active research on testing RDBMS, most of which employ crash or use metamorphic relations as the oracle. Although existing approaches can detect bugs in RDBMS, they are far from comprehensively evaluating the RDBMS's correctness (i.e., with respect to the semantics of SQL). In this work, we propose a method to test the semantic conformance of RDBMS i.e., whether its behavior respects the intended semantics of SQL. Specifically, we have formally defined the semantics of SQL and implemented them in Prolog. Then, the Prolog implementation serves as the reference RDBMS, enabling differential testing on existing RDBMS. We applied our approach to four widely-used and thoroughly tested RDBMSs, i.e., MySQL, TiDB, SQLite, and DuckDB. In total, our approach uncovered 19 bugs and 11 inconsistencies, which are all related to violating the SQL specification or missing/unclear specification, thereby demonstrating the effectiveness and applicability of our approach.