Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curse of scale-freeness: Intractability of large-scale optimization with multi-start methods

Hiroyuki Masuyama, Hiroshige Dan, Shunji Umetani

TL;DR

The paper formalizes the intractability of large-scale optimization with multi-start methods using Extreme Value Theory, deriving power-law relations for the evolution of the best empirical objective value and its relative gap. It proves scale-free decay in the expected relative gap, interprets this as a Zeno-like curse, and demonstrates via TSP experiments that overcoming the curse requires exponential acceleration beyond RMS. The results show that even strong random perturbation strategies like ILS struggle to beat the power-law decay, highlighting a fundamental limitation tied to tail behavior of solution quality. Practically, the work suggests shifting toward stopping criteria and estimation of expected improvement rates to manage computational effort in large-scale optimization problems.

Abstract

This paper investigates the intractability of large-scale optimization with multi-start methods. For the theoretical performance analysis, we focus on random multi-start (RMS), which is one of the representative multi-start methods, including RMS local search and greedy randomized adaptive search procedure (GRASP). Our primary theoretical contribution is to derive, by using extreme value theory, power-law formulas for the two quantities: (i) the expected improvement rate of the best empirical objective value (EOV); (ii) the expected relative gap between the best EOV and the supremum of the EOVs. Notably, the expected relative gap exhibits scale-freeness as a function of the number of iterations. Consequently, the half-life of the expected relative gap is asymptotically proportional to the number of iterations executed by the RMS method. This result can be interpreted as the curse of scale-freeness -- a Zeno's paradox-like phenomenon -- expressed by the metaphor "Reaching for the goal makes it slip away." Through numerical experiments, we observe that several RMS algorithms applied to traveling salesman problems suffer from the curse of scale-freeness. Furthermore, we show that overcoming this curse requires a powerful local search algorithm with effective restart and diversification strategies that exponentially accelerate solution improvement relative to the RMS method.

Curse of scale-freeness: Intractability of large-scale optimization with multi-start methods

TL;DR

The paper formalizes the intractability of large-scale optimization with multi-start methods using Extreme Value Theory, deriving power-law relations for the evolution of the best empirical objective value and its relative gap. It proves scale-free decay in the expected relative gap, interprets this as a Zeno-like curse, and demonstrates via TSP experiments that overcoming the curse requires exponential acceleration beyond RMS. The results show that even strong random perturbation strategies like ILS struggle to beat the power-law decay, highlighting a fundamental limitation tied to tail behavior of solution quality. Practically, the work suggests shifting toward stopping criteria and estimation of expected improvement rates to manage computational effort in large-scale optimization problems.

Abstract

This paper investigates the intractability of large-scale optimization with multi-start methods. For the theoretical performance analysis, we focus on random multi-start (RMS), which is one of the representative multi-start methods, including RMS local search and greedy randomized adaptive search procedure (GRASP). Our primary theoretical contribution is to derive, by using extreme value theory, power-law formulas for the two quantities: (i) the expected improvement rate of the best empirical objective value (EOV); (ii) the expected relative gap between the best EOV and the supremum of the EOVs. Notably, the expected relative gap exhibits scale-freeness as a function of the number of iterations. Consequently, the half-life of the expected relative gap is asymptotically proportional to the number of iterations executed by the RMS method. This result can be interpreted as the curse of scale-freeness -- a Zeno's paradox-like phenomenon -- expressed by the metaphor "Reaching for the goal makes it slip away." Through numerical experiments, we observe that several RMS algorithms applied to traveling salesman problems suffer from the curse of scale-freeness. Furthermore, we show that overcoming this curse requires a powerful local search algorithm with effective restart and diversification strategies that exponentially accelerate solution improvement relative to the RMS method.
Paper Structure (18 sections, 9 theorems, 60 equations, 7 figures, 2 tables, 2 algorithms)

This paper contains 18 sections, 9 theorems, 60 equations, 7 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The fundamental assumption of EVT
  4. Regularly varying functions and their scale-freeness in EVT
  5. Power-law formulas behind the curse of scale-freeness
  6. Understanding the curse of scale-freenes
  7. The curse of scale-freeness in connection to the ratio of good solutions
  8. Intractability of overcoming the curse of scale-freeness
  9. Concluding remarks
  10. Proof of Theorem \ref{['thm_asymp-E[R_n(x)]-01']}
  11. Proofs of propositions
  12. Proof of Proposition \ref{['prop_scale-free-01']}
  13. Proof of Proposition \ref{['prop_power-law-decay']}
  14. Proof of Proposition \ref{['prop_exponential-type-decay']}
  15. Setup of numerical experiments
  16. ...and 3 more sections

Key Result

Proposition 2.2

[Haan06] Assumption assumpt_GEV holds if and only if each of the conditions (a) and (b) is satisfied:

Figures (7)

  • Figure 1: The RMS method as a random sampler from the set $\mathbb{S}_{\rm G}$ of good solutions.
  • Figure 2: Evolution of the relative gap between the best EOV and the optimal value for an RMS algorithm applied to 100 random TSP (Traveling Salesman Problem) instances. Figures \ref{['fig_RE-NN-LKH-normal']} and \ref{['fig_RE-NN-LKH-log']} show the same data, but Figure \ref{['fig_RE-NN-LKH-normal']} uses linear axes, whereas Figure \ref{['fig_RE-NN-LKH-log']} uses logarithmic axes.
  • Figure 5: Evolution of the relative gap of the best EOV over $10^6$ iterations for each of the six RMS algorithms listed in Table \ref{['tb_RMS_algorithms']}, applied to 100 random TSP instances. The blue, red, and green lines represent the 90th percentile, mean, and 10th percentile values, respectively, based on the results obtained from the 100 instances.
  • Figure 12: Good-solution-ratio function $r(\varepsilon)$ calculated from the data used to generate Figure \ref{['fig_RMS_1000cities']}.
  • Figure 19: Evolution of the relative gap of the best EOV over 100 random sets of $10^6$ iterations of the RA+LK RMS and ILS algorithms, respectively, for the five TSPLIB instances listed in Table \ref{['TSPLIB_instances']}.
  • ...and 2 more figures

Theorems & Definitions (22)

  • Remark 1.2
  • Remark 1.3
  • Proposition 2.2
  • Remark 2.3
  • Definition 2.4
  • Remark 2.5
  • Proposition 2.6
  • Remark 2.7
  • Proposition 2.8
  • Theorem 3.1
  • ...and 12 more