Finding XPath Bugs in XML Document Processors via Differential Testing

TL;DR

XML processors with XPath support can harbor logic bugs that produce incorrect results. We propose XPress, a differential-testing framework that generates XML documents and XPath expressions guided by a targeted node to obtain non-empty results, then compares results across multiple engines. In experiments on six mature processors, XPress discovered $27$ unique bugs (with $20$ fixed and $25$ confirmed), and demonstrated about a $2x$ speedup over naive random generation. The approach is simple, general, and integrates test cases into existing suites, suggesting it can substantially improve the robustness of XML processors and inform future XPath-related testing.

Abstract

Extensible Markup Language (XML) is a widely used file format for data storage and transmission. Many XML processors support XPath, a query language that enables the extraction of elements from XML documents. These systems can be affected by logic bugs, which are bugs that cause the processor to return incorrect results. In order to tackle such bugs, we propose a new approach, which we realized as a system called XPress. As a test oracle, XPress relies on differential testing, which compares the results of multiple systems on the same test input, and identifies bugs through discrepancies in their outputs. As test inputs, XPress generates both XML documents and XPath queries. Aiming to generate meaningful queries that compute non-empty results, XPress selects a so-called targeted node to guide the XPath expression generation process. Using the targeted node, XPress generates XPath expressions that reference existing context related to the targeted node, such as its tag name and attributes, while also guaranteeing that a predicate evaluates to true before further expanding the query. We tested our approach on six mature XML processors, BaseX, eXist-DB, Saxon, PostgreSQL, libXML2, and a commercial database system. In total, we have found 20 unique bugs in these systems, of which 25 have been verified by the developers, and 12 of which have been fixed. XPress is efficient, as it finds 12 unique bugs in BaseX in 24 hours, which is 2x as fast as naive random generation. We expect that the effectiveness and simplicity of our approach will help to improve the robustness of many XML processors.

Figures (9)

• Figure 1: Example XML and motivating example.
• Figure 2: Simplified structure of XPath expressions.
• Figure 3: Overview of the approach implemented in XPress.
• Figure 4: Incorrect optimization of comparison conditions.
• Figure 5: Arithmetic overflow in pre-check conditions.