Scalable Derivative-Free Optimization Algorithms with Low-Dimensional Subspace Techniques
Zaikun Zhang
TL;DR
This work addresses scalable derivative-free optimization for high-dimensional problems by employing a subspace framework that builds a low-dimensional search subspace from an approximate gradient and solves a subproblem within that subspace. By deriving global convergence and worst-case complexity bounds and detailing derivative-free construction via fully linear models built from interpolation, the paper provides practical strategies for scalable optimization with noisy function values up to $n=10^4$. The introduction of SPRIMA, a package extending NEWUOA with OptimIST and PRIMA, demonstrates substantial empirical gains: on moderate-dimensional CUTEst problems, derivative-free subspace methods outperform traditional NEWUOA, and on large-scale problems ($n=10^4$) they remain viable where gradient-based solvers like fminunc often stall. These results highlight the approach’s potential for high-dimensional engineering design and simulation optimization where function evaluations are expensive or imprecise.
Abstract
We re-introduce a derivative-free subspace optimization framework originating from Chapter 5 of the Ph.D. thesis [Z. Zhang, On Derivative-Free Optimization Methods, Ph.D. thesis, Chinese Academy of Sciences, Beijing, 2012] of the author under the supervision of Ya-xiang Yuan. At each iteration, the framework defines a (low-dimensional) subspace based on an approximate gradient, and then solves a subproblem in this subspace to generate a new iterate. We sketch the global convergence and worst-case complexity analysis of the framework, elaborate on its implementation, and present some numerical results on solving problems with dimensions as high as 10^4 using only inaccurate function values.