Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fail Faster: Staging and Fast Randomness for High-Performance PBT

Cynthia Richey, Joseph W. Cutler, Harrison Goldstein, Benjamin C. Pierce

TL;DR

This paper tackles a critical bottleneck in property-based testing: slow generators caused by abstraction-heavy DSLs and suboptimal randomness sources. It introduces Allegro, a multi-stage programming approach that eliminates DSL overhead, and demonstrates its effectiveness in OCaml (AllegrOCaml) and Scala 3 (ScAllegro). In controlled experiments, staging alone yields substantial speedups (up to $13\times$ in Scala and $7\times$ in OCaml), and faster randomness (CSplitMix) amplifies these gains, with some cases exceeding $12\times$ improvements and Bug-finding speedups reaching around $2.5$–$3.8\times$ on average. The results, validated by pointwise semantic preservation and Etna-based bug-finding benchmarks, show that combining staging with optimized randomness can dramatically accelerate PBT across languages, with practical implications for faster software verification and bug discovery; the authors also outline paths to extend Allegro to additional languages and RNG strategies.

Abstract

Property-based testing (PBT) relies on generators for random test cases, often constructed using embedded domain specific languages, which provide expressive combinators for building and composing generators. The effectiveness of PBT depends critically on the speed of these generators. However, careful measurements show that the generator performance of widely used PBT libraries falls well short of what is possible, due principally to (1) the abstraction overhead of their combinator-heavy style and (2) suboptimal sources of randomness. We characterize, quantify, and address these bottlenecks. To eliminate abstraction overheads, we propose a technique based on multi-stage programming, dubbed Allegro. We apply this technique to leading generator libraries in OCaml and Scala 3, significantly improving performance. To quantify the performance impact of the randomness source, we carry out a controlled experiment, replacing the randomness in the OCaml PBT library with an optimized version. Both interventions exactly preserve the semantics of generators, enabling precise, pointwise comparisons. Together, these improvements find bugs up to $13\times$ faster.

Fail Faster: Staging and Fast Randomness for High-Performance PBT

TL;DR

This paper tackles a critical bottleneck in property-based testing: slow generators caused by abstraction-heavy DSLs and suboptimal randomness sources. It introduces Allegro, a multi-stage programming approach that eliminates DSL overhead, and demonstrates its effectiveness in OCaml (AllegrOCaml) and Scala 3 (ScAllegro). In controlled experiments, staging alone yields substantial speedups (up to in Scala and in OCaml), and faster randomness (CSplitMix) amplifies these gains, with some cases exceeding improvements and Bug-finding speedups reaching around – on average. The results, validated by pointwise semantic preservation and Etna-based bug-finding benchmarks, show that combining staging with optimized randomness can dramatically accelerate PBT across languages, with practical implications for faster software verification and bug discovery; the authors also outline paths to extend Allegro to additional languages and RNG strategies.

Abstract

Property-based testing (PBT) relies on generators for random test cases, often constructed using embedded domain specific languages, which provide expressive combinators for building and composing generators. The effectiveness of PBT depends critically on the speed of these generators. However, careful measurements show that the generator performance of widely used PBT libraries falls well short of what is possible, due principally to (1) the abstraction overhead of their combinator-heavy style and (2) suboptimal sources of randomness. We characterize, quantify, and address these bottlenecks. To eliminate abstraction overheads, we propose a technique based on multi-stage programming, dubbed Allegro. We apply this technique to leading generator libraries in OCaml and Scala 3, significantly improving performance. To quantify the performance impact of the randomness source, we carry out a controlled experiment, replacing the randomness in the OCaml PBT library with an optimized version. Both interventions exactly preserve the semantics of generators, enabling precise, pointwise comparisons. Together, these improvements find bugs up to faster.

Paper Structure

This paper contains 24 sections, 1 equation, 18 figures.

Table of Contents

  1. Introduction
  2. What are Generator Libraries, and Why are They Slow?
  3. Abstraction Overhead of Generator DSLs
  4. Overhead of Choice Combinators
  5. Inefficient Randomness Libraries
  6. How Can I Make My Generator Library Faster?
  7. Background: Multi-Stage Programming
  8. Staging eDSLs
  9. Staging in MetaOCaml
  10. Design of a Staged Generator DSL
  11. Staging Combinators
  12. Let-Insertion and Effect Ordering
  13. CodeCPS and a Monad Instance
  14. Recursive Generators
  15. Staging Type-Derived Generators
  16. ...and 9 more sections

Figures (18)

  • Figure 1: Some functions from the API of the Base_quickcheck generator DSL.
  • Figure 2: Simple monadic generators for a pair of ordered integers.
  • Figure 3: A generator using a variety of convenience functions.
  • Figure 4: Internals of a Monadic Generator eDSL
  • Figure 5: Microbenchmarks of generators in Base_quickcheck and ScalaCheck. Average over 10,000 generations with random seeds and a fixed size.
  • ...and 13 more figures