Distinguishability-guided Test Program Generation for WebAssembly Runtime Performance Testing

TL;DR

This work tackles the lack of high-quality test programs for Wasm runtime performance testing by introducing WarpGen, a history-driven, distinguishability-guided test program generator. WarpGen extracts operators from historical abnormal cases and synthesizes new test programs by inserting these operators into seed programs, guided by a dist score that measures how well a program reveals abnormal performance across Wasm runtimes. The approach efficiently yields high-quality test programs and outperforms random generation and non-guided variants, uncovering seven previously unknown performance issues across four Wasm runtimes. Case studies and extensive evaluation demonstrate WarpGen’s practical impact for improving reliability and performance in server-side Wasm systems.

Abstract

WebAssembly (Wasm) is a binary instruction format designed as a portable compilation target, which has been widely used on both the web and server sides in recent years. As high performance is a critical design goal of Wasm, it is essential to conduct performance testing for Wasm runtimes. However, existing research on Wasm runtime performance testing still suffers from insufficient high-quality test programs. To solve this problem, we propose a novel test program generation approach WarpGen. It first extracts code snippets from historical issue-triggering test programs as initial operators, then inserts an operator into a seed program to synthesize a new test program. To verify the quality of generated programs, we propose an indicator called distinguishability, which refers to the ability of a test program to distinguish abnormal performance of specific Wasm runtimes. We apply WarpGen for performance testing on four Wasm runtimes and verify its effectiveness compared with baseline approaches. In particular, WarpGen has identified seven new performance issues in three Wasm runtimes.

Figures (5)

• Figure 1: The workflow of Wasm compilation and execution.
• Figure 2: The framework of WarpGen.
• Figure 3: A looping operator and its pre/post-contexts. The type and positions of the variables are omitted.
• Figure 4: Comparison of top 20 dist score from different approaches.
• Figure 5: Test program related to Issue #7731.