Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predicting Software Performance with Divide-and-Learn

Jingzhi Gong, Tao Chen

TL;DR

The paper tackles the challenge of predicting software performance in highly configurable systems where data sparsity and feature sparsity hinder learning. It introduces divide-and-learn (DaL), which partitions data into divisions via an extended CART clustering, trains a dedicated regularized DNN for each division, and assigns new configurations to the appropriate division using a Random Forest on pseudo-labeled data with SMOTE. Empirically, DaL achieves comparable or superior accuracy to state-of-the-art methods in 33 of 40 cases (up to 1.94x improvement), often requiring fewer training samples, and remains robust across different local models with the default rDNN offering the best performance. The work provides practical guidance on depth settings and demonstrates DaL’s flexibility and potential for real-world deployment, including open data and code resources for open science.

Abstract

Predicting the performance of highly configurable software systems is the foundation for performance testing and quality assurance. To that end, recent work has been relying on machine/deep learning to model software performance. However, a crucial yet unaddressed challenge is how to cater for the sparsity inherited from the configuration landscape: the influence of configuration options (features) and the distribution of data samples are highly sparse. In this paper, we propose an approach based on the concept of 'divide-and-learn', dubbed DaL. The basic idea is that, to handle sample sparsity, we divide the samples from the configuration landscape into distant divisions, for each of which we build a regularized Deep Neural Network as the local model to deal with the feature sparsity. A newly given configuration would then be assigned to the right model of division for the final prediction. Experiment results from eight real-world systems and five sets of training data reveal that, compared with the state-of-the-art approaches, DaL performs no worse than the best counterpart on 33 out of 40 cases (within which 26 cases are significantly better) with up to 1.94x improvement on accuracy; requires fewer samples to reach the same/better accuracy; and producing acceptable training overhead. Practically, DaL also considerably improves different global models when using them as the underlying local models, which further strengthens its flexibility. To promote open science, all the data, code, and supplementary figures of this work can be accessed at our repository: https://github.com/ideas-labo/DaL.

Predicting Software Performance with Divide-and-Learn

TL;DR

The paper tackles the challenge of predicting software performance in highly configurable systems where data sparsity and feature sparsity hinder learning. It introduces divide-and-learn (DaL), which partitions data into divisions via an extended CART clustering, trains a dedicated regularized DNN for each division, and assigns new configurations to the appropriate division using a Random Forest on pseudo-labeled data with SMOTE. Empirically, DaL achieves comparable or superior accuracy to state-of-the-art methods in 33 of 40 cases (up to 1.94x improvement), often requiring fewer training samples, and remains robust across different local models with the default rDNN offering the best performance. The work provides practical guidance on depth settings and demonstrates DaL’s flexibility and potential for real-world deployment, including open data and code resources for open science.

Abstract

Predicting the performance of highly configurable software systems is the foundation for performance testing and quality assurance. To that end, recent work has been relying on machine/deep learning to model software performance. However, a crucial yet unaddressed challenge is how to cater for the sparsity inherited from the configuration landscape: the influence of configuration options (features) and the distribution of data samples are highly sparse. In this paper, we propose an approach based on the concept of 'divide-and-learn', dubbed DaL. The basic idea is that, to handle sample sparsity, we divide the samples from the configuration landscape into distant divisions, for each of which we build a regularized Deep Neural Network as the local model to deal with the feature sparsity. A newly given configuration would then be assigned to the right model of division for the final prediction. Experiment results from eight real-world systems and five sets of training data reveal that, compared with the state-of-the-art approaches, DaL performs no worse than the best counterpart on 33 out of 40 cases (within which 26 cases are significantly better) with up to 1.94x improvement on accuracy; requires fewer samples to reach the same/better accuracy; and producing acceptable training overhead. Practically, DaL also considerably improves different global models when using them as the underlying local models, which further strengthens its flexibility. To promote open science, all the data, code, and supplementary figures of this work can be accessed at our repository: https://github.com/ideas-labo/DaL.
Paper Structure (37 sections, 7 equations, 5 figures, 5 tables, 1 algorithm)

This paper contains 37 sections, 7 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Problem Formulation
  4. Sparsity in Software Performance Learning
  5. Divide-and-Learn for Performance Prediction
  6. Dividing
  7. Training
  8. Predicting
  9. Trade-off with the Number of Divisions
  10. Experiment Setup
  11. Research Questions
  12. Subject Systems
  13. Metric and Statistical Validation
  14. Accuracy
  15. Statistical Test
  16. ...and 22 more sections

Figures (5)

  • Figure 1: Projection of configurations in the landscape with respect to the performance and two most important options (the divisions are circled).
  • Figure 2: The architecture of DaL.
  • Figure 3: Projection of CART for VP8 showing the possible divisions with different colors under alternative depth $d$.
  • Figure 4: The median MRE (), its IQR (area), and the average smallest training size of the divisions () achieved by DaL under different depths ($d$ values)/number of divisions over all systems and 30 runs. $r=1$ means rank 1 in the Scott-Knott test on MRE. The best-ranked $d$ is marked as $*$.
  • Figure 5: Example run of the actual and predicted performance by DaL and DeepPerf for VP8.