Testing the Unknown: A Framework for OpenMP Testing via Random Program Generation

TL;DR

By generating 1,800 OpenMP tests, this work finds various performance anomalies and correctness issues when they apply them to three OpenMP implementations: GCC, Clang, and Intel.

Abstract

We present a randomized differential testing approach to test OpenMP implementations. In contrast to previous work that manually creates dozens of verification and validation tests, our approach is able to randomly generate thousands of tests, exposing OpenMP implementations to a wide range of program behaviors. We represent the space of possible random OpenMP tests using a grammar and implement our method as an extension of the Varity program generator. By generating 1,800 OpenMP tests, we find various performance anomalies and correctness issues when we apply it to three OpenMP implementations: GCC, Clang, and Intel. We also present several case studies that analyze the anomalies and give more details about the classes of tests that our approach creates.

Figures (11)

• Figure 1: Overview of our approach. Step (a) generates program tests and inputs; in step (b) we compile the test with multiple OpenMP compilers---the goal is to find a bug in any of the OpenMP implementations; (in step (c) we run the tests and gather numerical results and execution time; in step (d) we compare the numerical results and executions times and identify bugs by finding outliers. The figure highlights am anomaly in the test produced by the OpenMP implementation 3, in which the execution time is significantly different from the execution times of the other tests (generated by the other OpenMP implementations).
• Figure 2: Random test that induces a performance anomaly in Clang.
• Figure 3: High-level specification of the grammar for the random test programs. <fp-type> supports {float, double}, <assignment-op> supports {=, +=, -=, *=, /=}, <op> supports {+, -, *, /}, and <bool-op> supports {<, >, ==, !=, >=, <=}. <fp-numeral> is a constant, e.g., $1.23\text{e+}4$. <reduction-op> supports {+, *}.
• Figure 4: Example of how parameters control the code generation.
• Figure 5: Example of production rules representing the code generated with an if-condition block, and two assignments. One of the assignments (line 2) has a arithmetic expression with several terms.