Faster Configuration Performance Bug Testing with Neural Dual-level Prioritization

TL;DR

CPBugs pose a major challenge for configurable software due to large configuration spaces and unclear testing Oracle. The authors introduce Neural Dual-level Prioritization (NDP), which employs two RoBERTa models to predict CPBug types and CPBug-related option probabilities, enabling prioritization at both the option level and the numeric search depth. Empirical results show NDP achieves superior CPBug-type prediction (about 87% of type-metrics) and delivers substantial efficiency gains (up to 1.73x option-testing speedups and up to 88.88x numeric-search speedups), including discovery of new CPBugs. The work advances automated, data-driven CPBug testing and provides a publicly available repository for replication and extension.

Abstract

As software systems become more complex and configurable, more performance problems tend to arise from the configuration designs. This has caused some configuration options to unexpectedly degrade performance which deviates from their original expectations designed by the developers. Such discrepancies, namely configuration performance bugs (CPBugs), are devastating and can be deeply hidden in the source code. Yet, efficiently testing CPBugs is difficult, not only due to the test oracle is hard to set, but also because the configuration measurement is expensive and there are simply too many possible configurations to test. As such, existing testing tools suffer from lengthy runtime or have been ineffective in detecting CPBugs when the budget is limited, compounded by inaccurate test oracle. In this paper, we seek to achieve significantly faster CPBug testing by neurally prioritizing the testing at both the configuration option and value range levels with automated oracle estimation. Our proposed tool, dubbed NDP, is a general framework that works with different heuristic generators. The idea is to leverage two neural language models: one to estimate the CPBug types that serve as the oracle while, more vitally, the other to infer the probabilities of an option being CPBug-related, based on which the options and the value ranges to be searched can be prioritized. Experiments on several widely-used systems of different versions reveal that NDP can, in general, better predict CPBug type in 87% cases and find more CPBugs with up to 88.88x testing efficiency speedup over the state-of-the-art tools.

Figures (6)

• Figure 1: Workflow overview of NDP for CPBug testing.
• Figure 2: Illustrations of the search space bounded by different search depths for CPBug testing with NDP.
• Figure 3: Exampled kernel density function on the numeric options' probabilities of being CPBug-related for a system.
• Figure 4: Effectiveness of testing CPBugs over all options.
• Figure 5: Testing CPBugs over numeric options.