Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pushing the Limits of the Reactive Affine Shaker Algorithm to Higher Dimensions

Roberto Battiti, Mauro Brunato

TL;DR

The paper tackles the challenge of optimizing expensive functions in very high dimensions by proposing the Reactive Affine Shaker (RAS), a simple local-search heuristic that relies on an anisotropic, affine-updated search box and only uses success/failure feedback rather than actual function values. RAS updates a box around the current point with random direction vectors, expanding along successful directions via an affine transform $A=oldsymbol{I} + (\\rho-1) \\frac{\\boldsymbol{\\Delta}\\boldsymbol{\\Delta}^T}{\\|\\boldsymbol{\\Delta}\\|^2}$ and contracting after failures, allowing the search to align with promising directions even in high dimensions. The authors conduct an ablation study and compare RAS against state-of-the-art high-dimensional Bayesian optimization methods (e.g., TuRBO, SaasBO, Alebo, HeSBO) and CMA-ES on diverse benchmarks (including BAxUS), showing that while RAS often matches or approaches BO performance, its strengths lie in simplicity, robustness, and insights into the geometry of high-dimensional searches. The work suggests that RAS can serve as a strong baseline or be hybridized with BO to balance exploration and exploitation, and it opens avenues for parallel/multi-start extensions (e.g., M-RAS) and extensions to mixed/combinatorial spaces.

Abstract

Bayesian Optimization (BO) for the minimization of expensive functions of continuous variables uses all the knowledge acquired from previous samples (${\boldsymbol x}_i$ and $f({\boldsymbol x}_i)$ values) to build a surrogate model based on Gaussian processes. The surrogate is then exploited to define the next point to sample, through a careful balance of exploration and exploitation. Initially intended for low-dimensional spaces, BO has recently been modified and used also for very large-dimensional spaces (up to about one thousand dimensions). In this paper we consider a much simpler algorithm, called "Reactive Affine Shaker" (RAS). The next sample is always generated with a uniform probability distribution inside a parallelepiped (the "box"). At each iteration, the form of the box is adapted during the search through an affine transformation, based only on the point $\boldsymbol x$ position and on the success or failure in improving the function. The function values are therefore not used directly to modify the search area and to generate the next sample. The entire dimensionality is kept (no active subspaces). Despite its extreme simplicity and its use of only stochastic local search, surprisingly the produced results are comparable to and not too far from the state-of-the-art results of high-dimensional versions of BO, although with some more function evaluations. An ablation study and an analysis of probability distribution of directions (improving steps and prevailing box orientation) in very large-dimensional spaces are conducted to understand more about the behavior of RAS and to assess the relative importance of the algorithmic building blocks for the final results.

Pushing the Limits of the Reactive Affine Shaker Algorithm to Higher Dimensions

TL;DR

The paper tackles the challenge of optimizing expensive functions in very high dimensions by proposing the Reactive Affine Shaker (RAS), a simple local-search heuristic that relies on an anisotropic, affine-updated search box and only uses success/failure feedback rather than actual function values. RAS updates a box around the current point with random direction vectors, expanding along successful directions via an affine transform and contracting after failures, allowing the search to align with promising directions even in high dimensions. The authors conduct an ablation study and compare RAS against state-of-the-art high-dimensional Bayesian optimization methods (e.g., TuRBO, SaasBO, Alebo, HeSBO) and CMA-ES on diverse benchmarks (including BAxUS), showing that while RAS often matches or approaches BO performance, its strengths lie in simplicity, robustness, and insights into the geometry of high-dimensional searches. The work suggests that RAS can serve as a strong baseline or be hybridized with BO to balance exploration and exploitation, and it opens avenues for parallel/multi-start extensions (e.g., M-RAS) and extensions to mixed/combinatorial spaces.

Abstract

Bayesian Optimization (BO) for the minimization of expensive functions of continuous variables uses all the knowledge acquired from previous samples ( and values) to build a surrogate model based on Gaussian processes. The surrogate is then exploited to define the next point to sample, through a careful balance of exploration and exploitation. Initially intended for low-dimensional spaces, BO has recently been modified and used also for very large-dimensional spaces (up to about one thousand dimensions). In this paper we consider a much simpler algorithm, called "Reactive Affine Shaker" (RAS). The next sample is always generated with a uniform probability distribution inside a parallelepiped (the "box"). At each iteration, the form of the box is adapted during the search through an affine transformation, based only on the point position and on the success or failure in improving the function. The function values are therefore not used directly to modify the search area and to generate the next sample. The entire dimensionality is kept (no active subspaces). Despite its extreme simplicity and its use of only stochastic local search, surprisingly the produced results are comparable to and not too far from the state-of-the-art results of high-dimensional versions of BO, although with some more function evaluations. An ablation study and an analysis of probability distribution of directions (improving steps and prevailing box orientation) in very large-dimensional spaces are conducted to understand more about the behavior of RAS and to assess the relative importance of the algorithmic building blocks for the final results.

Paper Structure

This paper contains 14 sections, 8 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Bayesian Optimization for high-dimensional problems
  3. The RAS heuristic
  4. Implementation of the anisotropic search box
  5. Experimental Results
  6. Behavior of the search box
  7. Comparison with state-of-the-art heuristics
  8. Ablation study
  9. Isotropic search region
  10. Single-shot
  11. Conclusion
  12. Supplemental material --- Finding improving directions in high dimensions
  13. Random directions tend to be mutually orthogonal
  14. Improving moves tend to be scarce

Figures (8)

  • Figure 1: The RAS algorithm
  • Figure 2: Evolution of the search box $\mathcal{R}$ under transformation \ref{['eq:affine']}: from the current shape (center), the base vectors $\boldsymbol b_i$ are dilated in the direction of $\boldsymbol\Delta$ if the step succeeds (left), or contracted (right) if the step fails.
  • Figure 3: Evolution of the search box under transformation \ref{['eq:affine']}, 2D Rosenbrock function; on the right, a detailed view of the AB rectangle with the direction change.
  • Figure 4: Evolution of the search box under affine transformation wrt number of function evaluations for two test benchmarks, see the text for details.
  • Figure 5: Test results on high-dimensionality functions. First and third row: TuRBO, SaasBO, Alebo, HeSBO, CMA-ES, random search (reprinted from papenmeier2022increasing with permission). Second and fourth row: results on the same functions from 30 runs of RAS, plotted on comparable scales.
  • ...and 3 more figures