Less Noise, More Signal: DRR for Better Optimizations of SE Tasks

TL;DR

This work introduces the Dimensionality Reduction Ratio (DRR) as a predictor of when lightweight optimization suffices for SE analytics tasks. By estimating intrinsic dimensionality and challenging the Agrawal threshold, the authors demonstrate that many SE datasets possess high DRR, enabling simple methods like LITE to achieve performance comparable to heavy optimizers such as DEHB, with runtime dramatically reduced. The study spans 24 SE datasets and contrasts against non-SE baselines, showing SE problems can often be solved two orders of magnitude faster without loss of quality. The findings advocate dataset-aware optimization, caution against wholesale use of complex AI optimizers, and propose integrating DRR into practical SE analytics pipelines for adaptive method selection.

Abstract

SE analytics problems do not always need complex AI. Better and faster solutions can sometimes be obtained by matching the complexity of the problem to the complexity of the solution. This paper introduces the Dimensionality Reduction Ratio (DRR), a new metric for predicting when lightweight algorithms suffice. Analyzing SE optimization problems from software configuration to process decisions and open-source project health we show that DRR pinpoints "simple" tasks where costly methods like DEHB (a state-of-the-art evolutionary optimizer) are overkill. For high-DRR problems, simpler methods can be just as effective and run two orders of magnitude faster.

Figures (7)

• Figure 1: Differences in the intrinsic dimensionality of SE and non-SE data. From agrawal2021simpler.
• Figure 2: According to Agrawal et al. agrawal2021simpler different algorithms work best at different intrinsic dimensionalities. Vertical dashed lines shows the median of the SE data from Figure \ref{['ag1']}.
• Figure 3: Intrinsic Dimensionality vs Original Dimensionality of the Table \ref{['data']} data. (Blue dots are SE and Red dots NON-SE)
• Figure 4: Shown here are the Table \ref{['data']} data sets, scored in the y-axis by Equation \ref{['drr1']} (and the data is spaced out across the x-axis for readability). For the blue points, in all cases, very simple optimizers (that use only 30 samples) perform as well as more complex optimizers (that require 3000 samples).
• Figure 5: A range of hyperparameter optimization methods. From bischl2023hyperparameter. This paper compares the methods shown in red.