Understanding and Finding JIT Compiler Performance Bugs

TL;DR

This paper proposes layered differential performance testing, a lightweight technique to automatically detect JIT compiler performance bugs, and implements it in a tool called Jittery, which discovered 12 previously unknown performance bugs in the Oracle HotSpot and Graal JIT compilers.

Abstract

Just-in-time (JIT) compilers are key components for many popular programming languages with managed runtimes (e.g., Java and JavaScript). JIT compilers perform optimizations and generate native code at runtime based on dynamic profiling data, to improve the execution performance of the running application. Like other software systems, JIT compilers might have software bugs, and prior work has developed a number of automated techniques for detecting functional bugs (i.e., generated native code does not semantically match that of the original code). However, no prior work has targeted JIT compiler performance bugs, which can cause significant performance degradation while an application is running. These performance bugs are challenging to detect due to the complexity and dynamic nature of JIT compilers. In this paper, we present the first work on demystifying JIT performance bugs. First, we perform an empirical study across four popular JIT compilers for Java and JavaScript. Our manual analysis of 191 bug reports uncovers common triggers of performance bugs, patterns in which these bugs manifest, and their root causes. Second, informed by these insights, we propose layered differential performance testing, a lightweight technique to automatically detect JIT compiler performance bugs, and implement it in a tool called Jittery. We incorporate practical optimizations into Jittery such as test prioritization, which reduces testing time by 92.40% without compromising bug-detection capability, and automatic filtering of false-positives and duplicates, which substantially reduces manual inspection effort. Using Jittery, we discovered 12 previously unknown performance bugs in the Oracle HotSpot and Graal JIT compilers, with 11 confirmed and 6 fixed by developers.

Figures (9)

• Figure 1: High-level workflow of running a program through JIT compilation, which happens while the program is running. Commonly, JIT compilers optimize code snippets that are frequently executed.
• Figure 2: Example of a null-check elimination speculative optimization in HotSpot: (a) a common null check code pattern in Java source code; (b) the pseudocode generated by HotSpot with null-check eliminated. The optimization removes the null check and uses an implicit guard when speculation fails.
• Figure 3: Example of type specialization for add operator speculative optimization in V8: (a) add operator is overloaded for many different types in JavaScript source code; (b) the pseudocode generated by V8 with type specialization based on profiling data. The optimization specializes the add operator for small integer addition and uses an explicit guard to ensure the assumption holds.
• Figure 4: Common transitions among execution levels (L0--L4) in HotSpot tiered compilation. Execution starts in the interpreter (L0), then typically moves to L2 or L3 based on queue/load heuristics; methods with sufficient profiling in L3 are promoted to L4, while trivial methods may end at L1.
• Figure 5: Case study of https://bugs.openjdk.org/browse/JDK-8280320: loop optimizations are missing during On-Stack Replacement (OSR) compilation. (a) A test case that triggers OSR compilation but is not properly optimized; (b) A fix that avoids skewing counter data during OSR compilation. The bug is caused by incorrectly set profiling data, which prevents the vectorization optimization in HotSpot.