QuickerCheck: Implementing and Evaluating a Parallel Run-Time for QuickCheck

TL;DR

QuickerCheck develops a parallel run-time for QuickCheck to accelerate property-based testing by running tests and shrinking in parallel across cores. The approach preserves the QuickCheck API while introducing a multi-threaded testing loop, work-stealing, and a parallel shrinking loop with deterministic and greedy strategies. Evaluations across six benchmarks show substantial speedups on slower properties and mixed results for shrinking, highlighting when parallel shrinking helps and when its overhead dominates. The work demonstrates practical impact by reducing programmer waiting time and outlines directions for broader integration and alternative size strategies.

Abstract

This paper introduces a new parallel run-time for QuickCheck, a Haskell library and EDSL for specifying and randomly testing properties of programs. The new run-time can run multiple tests for a single property in parallel, using the available cores. Moreover, if a counterexample is found, the run-time can also shrink the test case in parallel, implementing a parallel search for a locally minimal counterexample. Our experimental results show a 3--9$\times$ speed-up for testing QuickCheck properties on a variety of heavy-weight benchmark problems. We also evaluate two different shrinking strategies; deterministic shrinking, which guarantees to produce the same minimal test case as standard sequential shrinking, and greedy shrinking, which does not have this guarantee but still produces a locally minimal test case, and is faster in practice.

Figures (20)

• Figure 1: An illustration of how size grows as more and more tests are passed, and to which worker they are assigned. In order to get a fair workload for the concurrent testers a stride is applied when computing sizes.
• Figure 2: The three figures above illustrate how the search for a minimized counterexample happened. The dotted line represents the final path to the local minimum, green boxes are candidate counterexamples that turned out to not be new counterexamples, and red boxes are counterexamples that still falsified the property. Grey boxes are candidate counterexamples that were never evaluated.
• Figure 3: A high-level description of the internal testing loop. The loop begins by generating input and then invoking the property. After this, the loop inspects the outcome before it either reports having found a counterexample, or loops back to repeat all steps. The bottom box and all arrows are part of the internal testing loop, while the top two boxes are defined by the user.
• Figure 4: The performance of sequential QuickerCheck compared to that of QuickCheck. A number of 1 means that there was no difference in performance, whereas a number less than 1 indicates that QuickerCheck was faster than QuickCheck (e.g. 0.5 shows that QuickerCheck finished in half the time). A number greater than 1 indicates that QuickerCheck was slower than QuickCheck.
• Figure 5: The acquired speedup relative to the sequential execution time when running tests.