Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards the Systematic Testing of Regular Expression Engines

Berk Çakar, Dongyoon Lee, James C. Davis

TL;DR

ReTest is described, a framework that systematically tests regular expression engines by combining grammar-aware fuzzing for high code coverage with metamorphic testing to generate dialect-independent test oracles and the preliminary evaluation on PCRE shows that ReTest achieves 3x higher edge coverage than existing fuzzing approaches and has identified three new memory safety defects.

Abstract

Software engineers use regular expressions (regexes) across a wide range of domains and tasks. To support regexes, software projects must integrate a regex engine, whether provided natively by the language runtime (e.g., Python's re) or included as an external dependency (e.g., PCRE). However, these engines may contain bugs and introduce vulnerabilities. A common strategy for testing regex engines involves differential testing -- comparing outputs across different implementations. However, this approach is concerning because regex syntax and semantics vary significantly between dialects (e.g., POSIX vs. PCRE). Fuzzing is also utilized to ease testing of feature-rich regex implementations to expose defects, but naive byte-level mutations generate syntactically invalid inputs that exercise only parsing logic, not matching internals. In this work, we describe our progress towards ReTest, a framework that systematically tests regular expression engines by combining grammar-aware fuzzing for high code coverage with metamorphic testing to generate dialect-independent test oracles. So far, we have surveyed testing practices across 22 regex engines, analyzed 1,007 regex engine bugs and 156 CVEs to characterize failure modes, and curated 16 metamorphic relations for regexes derived from Kleene algebra. Our preliminary evaluation on PCRE shows that ReTest achieves 3x higher edge coverage than existing fuzzing approaches and has identified three new memory safety defects. We conclude by describing our next steps toward our ultimate goal: helping regex engine developers identify bugs without depending on a consistent cross-implementation standard.

Towards the Systematic Testing of Regular Expression Engines

TL;DR

ReTest is described, a framework that systematically tests regular expression engines by combining grammar-aware fuzzing for high code coverage with metamorphic testing to generate dialect-independent test oracles and the preliminary evaluation on PCRE shows that ReTest achieves 3x higher edge coverage than existing fuzzing approaches and has identified three new memory safety defects.

Abstract

Software engineers use regular expressions (regexes) across a wide range of domains and tasks. To support regexes, software projects must integrate a regex engine, whether provided natively by the language runtime (e.g., Python's re) or included as an external dependency (e.g., PCRE). However, these engines may contain bugs and introduce vulnerabilities. A common strategy for testing regex engines involves differential testing -- comparing outputs across different implementations. However, this approach is concerning because regex syntax and semantics vary significantly between dialects (e.g., POSIX vs. PCRE). Fuzzing is also utilized to ease testing of feature-rich regex implementations to expose defects, but naive byte-level mutations generate syntactically invalid inputs that exercise only parsing logic, not matching internals. In this work, we describe our progress towards ReTest, a framework that systematically tests regular expression engines by combining grammar-aware fuzzing for high code coverage with metamorphic testing to generate dialect-independent test oracles. So far, we have surveyed testing practices across 22 regex engines, analyzed 1,007 regex engine bugs and 156 CVEs to characterize failure modes, and curated 16 metamorphic relations for regexes derived from Kleene algebra. Our preliminary evaluation on PCRE shows that ReTest achieves 3x higher edge coverage than existing fuzzing approaches and has identified three new memory safety defects. We conclude by describing our next steps toward our ultimate goal: helping regex engine developers identify bugs without depending on a consistent cross-implementation standard.
Paper Structure (43 sections, 4 figures, 3 tables, 2 algorithms)

This paper contains 43 sections, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Regex Engines
  4. Research on Regex Engine Testing
  5. Metamorphic Testing
  6. Research Overview
  7. Phase 1: Empirical Investigation
  8. Phase 2: Design and Evaluation of the ReTest Framework
  9. Testing Practices in Regex Engines
  10. Methodology
  11. The Testing Challenge
  12. Methodology
  13. Observations
  14. The Oracle Problem
  15. Developer-Defined Oracles
  16. ...and 28 more sections

Figures (4)

  • Figure 1: Distribution of bug types across 11 third-party regex engines. Semantic bugs form a plurality (35.3%) of all bugs.
  • Figure 2: Distribution of CVE types across regex engines.
  • Figure 3: Overview of the ReTest framework. The input generation module seeds fuzzing and maintains a subtree pool for grammar-aware AST mutations. The fuzzing loop generates test cases and tracks edge coverage to guide mutation. The oracle suite combines runtime sanitizers for crash detection with metamorphic testing for semantic bug detection.
  • Figure 4: Edge coverage over time for three strategies on PCRE v8.45.