Progressive Power Homotopy for Non-convex Optimization
Chen Xu
TL;DR
Prog-PowerHP introduces a single-loop stochastic optimization framework that pairs a progressive power transformation with Gaussian smoothing to solve non-convex max problems. It constructs the surrogate $F_{N,\sigma}(\bm{\mu})=\mathbb{E}_{\bm{w}\sim\mathcal{N}(\bm{\mu},\sigma^2I_d),\bm{x}\sim\mathcal{D}}[e^{N f_{\bm{w}}(\bm{x})}]$ and performs gradient updates while progressively increasing $N$ and decreasing $\sigma$, with theoretical guarantees that the method converges to a small neighborhood of the global optimum and a nearly optimal iteration complexity of $\mathcal{O}(d^2\varepsilon^{-2})$. Empirically, Prog-PowerHP excels in phase retrieval near the information-theoretic limit and in under-parameterized two-layer networks, indicating superior landscape navigation in cluttered non-convex regions compared to standard first-order and homotopy-based methods. The paper combines rigorous convergence analysis with practical demonstrations, highlighting the robustness of the approach in data-scarce and capacity-constrained regimes. Overall, Prog-PowerHP offers a principled, efficient route to approximate global optimization in challenging stochastic non-convex settings with potential broad impact on learning systems under resource constraints.
Abstract
We propose a novel first-order method for non-convex optimization of the form $\max_{\bm{w}\in\mathbb{R}^d}\mathbb{E}_{\bm{x}\sim\mathcal{D}}[f_{\bm{w}}(\bm{x})]$, termed Progressive Power Homotopy (Prog-PowerHP). The method applies stochastic gradient ascent to a surrogate objective obtained by first performing a power transformation and then Gaussian smoothing, $F_{N,σ}(\bmμ):=\mathbb{E}_{\bm{w}\sim\mathcal{N}(\bmμ,σ^2I_d),\bm{x}\sim\mathcal{D}}[e^{Nf_w(\bm{x})}]$, while progressively increasing the power parameter $N$ and decreasing the smoothing scale $σ$ along the optimization trajectory. We prove that, under mild regularity conditions, Prog-PowerHP converges to a small neighborhood of the global optimum with an iteration complexity scaling nearly as $O(d^2\varepsilon^{-2})$. Empirically, Prog-PowerHP demonstrates clear advantages in phase retrieval when the samples-to-dimension ratio approaches the information-theoretic limit, and in training two-layer neural networks in under-parameterized regimes. These results suggest that Prog-PowerHP is particularly effective for navigating cluttered non-convex landscapes where standard first-order methods struggle.