Optimal Restart Strategies for Parameter-dependent Optimization Algorithms
Lisa Schönenberger, Hans-Georg Beyer
TL;DR
This work examines restart strategies for parameter-dependent optimization algorithms where success requires surpassing an unknown threshold $\hat{\lambda}$. By classifying strategies into additive, multiplicative, and power-law types and introducing a loss function, the authors derive bounds and show that only multiplicative strategies yield a bounded relative loss in the asymptotic regime. They establish that the asymptotically optimal multiplicative factor is $\hat{\rho}=2$, with the $\mathcal{R}^\times$ and $\mathcal{R}^*$ variants achieving bounded performance and, for $\mathcal{R}^\times$, a worst-case relative loss of 3. Conversely, additive $\mathcal{R}^+$ and power-law $\mathcal{R}^\#$ strategies are strictly unbounded in relative loss. The results provide a principled, parameter-independent guideline favoring multiplicative restarts for robust, near-optimal performance in the face of unknown $\hat{\lambda}$.
Abstract
This paper examines restart strategies for algorithms whose successful termination depends on an unknown parameter $λ$. After each restart, $λ$ is increased, until the algorithm terminates successfully. It is assumed that there is a unique, unknown, optimal value for $λ$. For the algorithm to run successfully, this value must be reached or surpassed. The key question is whether there exists an optimal strategy for selecting $λ$ after each restart taking into account that the computational costs (runtime) increases with $λ$. In this work, potential restart strategies are classified into parameter-dependent strategy types. A loss function is introduced to quantify the wasted computational cost relative to the optimal strategy. A crucial requirement for any efficient restart strategy is that its loss, relative to the optimal $λ$, remains bounded. To this end, upper and lower bounds of the loss are derived. Using these bounds it will be shown that not all strategy types are bounded. However, for a particular strategy type, where $λ$ is increased multiplicatively by a constant factor $λ$, the relative loss function is bounded. Furthermore, it will be demonstrated that within this strategy type, there exists an optimal value for $λ$ that minimizes the maximum relative loss. In the asymptotic limit, this optimal choice of $λ$ does not depend on the unknown optimal $λ$.