Search Budget in Multi-Objective Refactoring Optimization: a Model-Based Empirical Study

TL;DR

The paper analyzes the impact of a time-based search budget on the quality of solutions in model-based multi-objective refactoring optimization, comparing NSGA-II, SPEA2, and PESA2 across two benchmarks. Using Hypervolume ($HV$) against a reference front built from unbudgeted runs, it shows that tighter budgets significantly reduce front quality, with the algorithm choice often outweighing budget effects. PESA2 tends to produce the highest quality fronts within a budget, NSGA-II is the fastest enabling more evolutions, and SPEA2 is generally the slowest with weaker results. The findings guide designers in selecting algorithms based on budget constraints and emphasize the practical importance of budget-aware optimization in software model refactoring.

Abstract

Software model optimization is the task of automatically generate design alternatives, usually to improve quality aspects of software that are quantifiable, like performance and reliability. In this context, multi-objective optimization techniques have been applied to help the designer find suitable trade-offs among several non-functional properties. In this process, design alternatives can be generated through automated model refactoring, and evaluated on non-functional models. Due to their complexity, this type of optimization tasks require considerable time and resources, often limiting their application in software engineering processes. In this paper, we investigate the effects of using a search budget, specifically a time limit, to the search for new solutions. We performed experiments to quantify the impact that a change in the search budget may have on the quality of solutions. Furthermore, we analyzed how different genetic algorithms (i.e., NSGA-II, SPEA2, and PESA2) perform when imposing different budgets. We experimented on two case studies of different size, complexity, and domain. We observed that imposing a search budget considerably deteriorates the quality of the generated solutions, but the specific algorithm we choose seems to play a crucial role. From our experiments, NSGA-II is the fastest algorithm, while PESA2 generates solutions with the highest quality. Differently, SPEA2 is the slowest algorithm, and produces the solutions with the lowest quality.

Figures (3)

• Figure 1: The graphical representation of the approach. The approach takes as input: the set of all the available refactoring actions (Refactoring Actions), and the subject model (Initial UML Model). The Genetic Algorithm (i.e.,NSGA-II, SPEA2, and PESA2) randomly selects and combines refactoring actions (Refactoring) in order to build a set of Model Alternatives. #pas, #changes, reliability, and perfQ are the four objectives that drive the optimization process.
• Figure 2: Timelines of the number of evolutions performed by the algorithms in the different budget configurations, along with the achieved HV. Vertical bars show the average HV over 31 runs, and ticks represent the standard deviation from the mean.
• Figure 3: TTBS, and CoCoME Pareto frontiers obtained by the three algorithms when varying the time budget between 15, 30, and 60 minutes. The top-right corner is the optimal point, whereas the bottom-left corner is the worst one. Filled symbols are the three algorithms: NSGA-II is the gray squares, PESA2 is the blue circles, and SPEA2 is the green triangles.