Hyperellipsoid Density Sampling: Exploitative Sequences to Accelerate High-Dimensional Optimization

TL;DR

High-dimensional optimization suffers from the curse of dimensionality, making uniform sampling inefficient when optima lie in localized regions. The authors propose Hyperellipsoid Density Sampling (HDS), an adaptive, non-uniform sampling strategy that builds multiple hyperellipsoids from clustering and PCA of an initial Sobol QMC set, optionally biased by Gaussian weights, to aggressively explore promising regions. In experiments with differential evolution on the 29-function CEC2017 suite across $D$ up to 100, HDS yields statistically significant improvements over Sobol, with an overall geometric mean improvement of about $1.15\times$ and a modest run-time overhead, demonstrating the method’s robustness and practical value. The approach also introduces a flexible density-control mechanism that can be extended beyond optimization to non-parametric modeling and other data-driven tasks.

Abstract

The curse of dimensionality presents a pervasive challenge in optimization problems, with exponential expansion of the search space rapidly causing traditional algorithms to become inefficient or infeasible. An adaptive sampling strategy is presented to accelerate optimization in this domain as an alternative to uniform quasi-Monte Carlo (QMC) methods. This method, referred to as Hyperellipsoid Density Sampling (HDS), generates its sequences by defining multiple hyperellipsoids throughout the search space. HDS uses three types of unsupervised learning algorithms to circumvent high-dimensional geometric calculations, producing an intelligent, non-uniform sample sequence that exploits statistically promising regions of the parameter space and improves final solution quality in high-dimensional optimization problems. A key feature of the method is optional Gaussian weights, which may be provided to influence the sample distribution towards known locations of interest. This capability makes HDS versatile for applications beyond optimization, providing a focused, denser sample distribution where models need to concentrate their efforts on specific, non-uniform regions of the parameter space. The method was evaluated against Sobol, a standard QMC method, using differential evolution (DE) on the 29 CEC2017 benchmark test functions. The results show statistically significant improvements in solution geometric mean error (p < 0.05), with average performance gains ranging from 3% in 30D to 37% in 10D. This paper demonstrates the efficacy of HDS as a robust alternative to QMC sampling for high-dimensional optimization.

Figures (9)

• Figure 1: HDS sequence (yellow) and a comparison Sobol sequence (red) for 10,000 samples in 2D.
• Figure 2: 2D HDS sequence with weights applied towards $\mathbf{x=0.25}$, $\mathbf{y=0.25}$, for bounds of $\mathbf{[0,1]^D}$
• Figure 3: Dendrogram used to identify number of ellipsoids for a representative 2D sample sequence.
• Figure 4: HDS samples in 2D for number of ellipsoids $K=2$, using a reduced radial scaling factor to easily visualize the geometry.
• Figure 5: Sample distributions for HDS sequences in 2D (left) and 100D (right).