A Balanced Approach of Rapid Genetic Exploration and Surrogate Exploitation for Hyperparameter Optimization

TL;DR

An improved hyperparameter optimization (HPO) framework that integrates a linear surrogate model into the genetic algorithm (GA) allows for seamless integration of multiple optimization strategies, and the surrogate model significantly boosts its exploitation capabilities.

Abstract

This paper proposes a new method for hyperparameter optimization (HPO) that balances exploration and exploitation. While evolutionary algorithms (EAs) show promise in HPO, they often struggle with effective exploitation. To address this, we integrate a linear surrogate model into a genetic algorithm (GA), allowing for smooth integration of multiple strategies. This combination improves exploitation performance, achieving an average improvement of 1.89 percent (max 6.55 percent, min -3.45 percent) over existing HPO methods.

Figures (8)

• Figure 1: Rapid Genetic Exploration with Random Direction Hill Climbing Linear Exploitation Architecture
• Figure 2: The figure represents how hyperparameters are stored using the chromosome structure that records genetic information in a GA. Each gene stores individual hyperparameter values.
• Figure 3: This figure represents Rapid Genetic Operations, which generate more crossover and mutation in a single generation to produce progressive offspring.
• Figure 4: The Adaptive Mutation Probability Function (AMPF) represents the change in mutation probability as a function of slope in the fitness score history. It lowers the probability when the slope has a steep monotone, and increases the probability as the slope becomes smoother so that more mutations occur.( https://www.desmos.com/calculator/lhkqrgzqwn)
• Figure 5: This figure explains the structure of MISO, SIMO, and MIMO. SIMO and MIMO are designed as reversed structures in which the inputs and outputs of the Score and Hyperparameter are reversed. SIMO can estimate one hyperparameter by inputting the expected score. The MIMO model can estimate multiple optimal hyperparameters separately by adding noise, and this noise is used as an identifier in the model.