Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models

Lizhi Liao, Simon Eismann, Heng Li, Cor-Paul Bezemer, Diego Elias Costa, Andre van Hoorn, Weiyi Shang

TL;DR

The paper tackles the problem of late and expensive detection of performance regressions by integrating lightweight component-level performance data with system-level architectural models. It presents a four-step approach: collect local performance data, identify local deviations, propagate deviations to an architectural model via dependency graphs and maximum common-subgraph mapping, and detect end-to-end regressions by comparing model-based predictions. Through experiments on TeaStore and TrainTicket, the authors demonstrate that local deviations can be propagated to predict end-to-end performance changes, enabling early detection during development and providing a resource-efficient complement to full system testing. The results show strong alignment with system-level testing in many cases, with some limitations linked to model quality and workload variability, highlighting practical utility for DevOps workflows where testing resources are constrained.

Abstract

During software development, developers often make numerous modifications to the software to address existing issues or implement new features. However, certain changes may inadvertently have a detrimental impact on the overall system performance. To ensure that the performance of new software releases does not degrade, existing practices rely on system-level performance testing, such as load testing, or component-level performance testing to detect performance regressions. However, performance testing for the entire system is often expensive and time-consuming, posing challenges to adapting to the rapid release cycles common in modern DevOps practices. System-level performance testing cannot be conducted until the system is fully built and deployed. On the other hand, component-level testing focuses on isolated components, neglecting overall system performance and the impact of system workloads. In this paper, we propose a novel approach to early detection of performance regressions by bridging the local performance data generated by component-level testing and the system-level architectural models. Our approach uses local performance data to identify deviations at the component level, and then propagate these deviations to the architectural model. We then use the architectural model to predict regressions in the performance of the overall system. We evaluate our approach on two open-source benchmark systems and show that it can effectively detect end-to-end system performance regressions from local performance deviations with different intensities and under various system workloads. More importantly, our approach can detect regressions as early as in the development phase, in contrast to existing approaches that require the system to be fully built and deployed. Our approach is lightweight and can complement traditional system performance testing when testing resources are scarce.

Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models

TL;DR

The paper tackles the problem of late and expensive detection of performance regressions by integrating lightweight component-level performance data with system-level architectural models. It presents a four-step approach: collect local performance data, identify local deviations, propagate deviations to an architectural model via dependency graphs and maximum common-subgraph mapping, and detect end-to-end regressions by comparing model-based predictions. Through experiments on TeaStore and TrainTicket, the authors demonstrate that local deviations can be propagated to predict end-to-end performance changes, enabling early detection during development and providing a resource-efficient complement to full system testing. The results show strong alignment with system-level testing in many cases, with some limitations linked to model quality and workload variability, highlighting practical utility for DevOps workflows where testing resources are constrained.

Abstract

During software development, developers often make numerous modifications to the software to address existing issues or implement new features. However, certain changes may inadvertently have a detrimental impact on the overall system performance. To ensure that the performance of new software releases does not degrade, existing practices rely on system-level performance testing, such as load testing, or component-level performance testing to detect performance regressions. However, performance testing for the entire system is often expensive and time-consuming, posing challenges to adapting to the rapid release cycles common in modern DevOps practices. System-level performance testing cannot be conducted until the system is fully built and deployed. On the other hand, component-level testing focuses on isolated components, neglecting overall system performance and the impact of system workloads. In this paper, we propose a novel approach to early detection of performance regressions by bridging the local performance data generated by component-level testing and the system-level architectural models. Our approach uses local performance data to identify deviations at the component level, and then propagate these deviations to the architectural model. We then use the architectural model to predict regressions in the performance of the overall system. We evaluate our approach on two open-source benchmark systems and show that it can effectively detect end-to-end system performance regressions from local performance deviations with different intensities and under various system workloads. More importantly, our approach can detect regressions as early as in the development phase, in contrast to existing approaches that require the system to be fully built and deployed. Our approach is lightweight and can complement traditional system performance testing when testing resources are scarce.
Paper Structure (22 sections, 3 figures, 2 tables)

This paper contains 22 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Software performance testing
  4. Model-based performance analysis
  5. Approach
  6. Collecting local performance data
  7. Identifying local performance deviations
  8. Propagating local performance deviations to architectural model
  9. Detecting system performance regressions
  10. Evaluation Setup
  11. Subject systems
  12. Evaluation environment
  13. Performance regressions
  14. Component-level and system-level performance test design
  15. Architectural models
  16. ...and 7 more sections

Figures (3)

  • Figure 1: An example software system with two subsystems and its corresponding architectural model
  • Figure 2: An overview of our approach for system performance regression detection by bridging local performance data and architectural models
  • Figure 3: An illustrative example of propagating local performance deviations to architectural model (for subfigures (a), (b), and (c), the red boxes indicate components with performance deviations, while for subfigure (d), the red box indicates updated performance parameter in the architectural model)