Compiler Optimization Testing Based on Optimization-Guided Equivalence Transformations

TL;DR

Compiler optimization bugs pose reliability risks, and differential testing has limitations that can miss shared issues. This work introduces metamorphic testing guided by optimization opportunities to generate semantically equivalent test programs, then applies loop-based transformations to create transformed programs and checks for output discrepancies. The approach is instantiated with four loop transforms (unrolling, LICM, unswitching, fusion) and evaluated on GCC and LLVM, where five incorrect optimization bugs were detected. The results demonstrate the method's practicality and potential to strengthen compiler verification, with a replication package available for researchers.

Abstract

Compiler optimization techniques are inherently complex, and rigorous testing of compiler optimization implementation is critical. Recent years have witnessed the emergence of testing approaches for uncovering incorrect optimization bugs, but these approaches rely heavily on the differential testing mechanism, which requires comparing outputs across multiple compilers. This dependency gives rise to important limitations, including that (1) the tested functionality must be consistently implemented across all compilers and (2) shared bugs remain undetected. Thus, false alarms can be produced and significant manual efforts will be required. To overcome the limitations, we propose a metamorphic testing approach inspired by compiler optimizations. The approach is driven by how to maximize compiler optimization opportunities while effectively judging optimization correctness. Specifically, our approach first employs tailored code construction strategies to generate input programs that satisfy optimization conditions, and then applies various compiler optimization transformations to create semantically equivalent test programs. By comparing the outputs of pre- and post-transformation programs, this approach effectively identifies incorrect optimization bugs. We conducted a preliminary evaluation of this approach on GCC and LLVM, and we have successfully detected five incorrect optimization bugs at the time of writing. This result demonstrates the effectiveness and potential of our approach.

Figures (4)

• Figure 1: Workflow of our approach
• Figure 2: Examples of loop optimization transformation methods.
• Figure 3: A sample that triggers a GCC compiler bug.
• Figure 4: A sample that triggers a LLVM compiler bug.