Efficiency of higher-order algorithms for minimizing composite functions
Yassine Nabou, Ion Necoara
TL;DR
The paper develops a unified General Composite Higher-Order (GCHO) framework for minimizing f(x)=g(F(x))+h(x) where F and h may be nonsmooth or nonconvex. By replacing F with p-th order surrogates whose error has Lipschitz p-th derivatives, GCHO achieves descent and provable convergence to stationary points, with rates that adapt to KL geometry and convexity: nonconvex settings yield O(1/k^{p/(p+1)}) in first-order optimality and KL-driven linear/sublinear rates, while convex cases yield a global O(1/k^{p}) rate in function values. The adaptive variant (A-GCHO) uses a line-search over the surrogate’s regularization parameter to ensure finite termination and preserve convergence guarantees. Numerical experiments on min-max and least-squares reformulations demonstrate practical efficiency, with higher-order surrogates (p=2) often accelerating convergence and recovering global optima where standard solvers may fail. This work provides a broad, theoretically grounded framework for higher-order majorization-minimization in composite optimization, including nonsmooth and nonconvex scenarios, and connects to a wide range of applications such as minimax approximations and robust factorization.
Abstract
Composite minimization involves a collection of functions which are aggregated in a nonsmooth manner. It covers, as a particular case, smooth approximation of minimax games, minimization of max-type functions, and simple composite minimization problems, where the objective function has a nonsmooth component. We design a higher-order majorization algorithmic framework for fully composite problems (possibly nonconvex). Our framework replaces each component with a higher-order surrogate such that the corresponding error function has a higher-order Lipschitz continuous derivative. We present convergence guarantees for our method for composite optimization problems with (non)convex and (non)smooth objective function. In particular, we prove stationary point convergence guarantees for general nonconvex (possibly nonsmooth) problems and under Kurdyka-Lojasiewicz (KL) property of the objective function we derive improved rates depending on the KL parameter. For convex (possibly nonsmooth) problems we also provide sublinear convergence rates.