(Adaptive) Scaled gradient methods beyond locally Holder smoothness: Lyapunov analysis, convergence rate and complexity
Susan Ghaderi, Morteza Rahimi, Yves Moreau, Masoud Ahookhosh
TL;DR
This work studies unconstrained convex minimization where the gradient is locally, rather than globally, Hölder continuous. By developing a local-smoothness framework, it analyzes the Scaled Gradient Algorithm (SGA) and proves global convergence with explicit complexity bounds, plus linear convergence under local strong convexity or KL inequality. It further introduces AdaSGA, an adaptive variant that bypasses the need for a known global Lipschitz modulus via Lyapunov-based analysis, and shows global convergence with local linear rates under the same extra conditions. The results extend classical gradient methods to broader smoothness regimes and provide practical schemes for step-size and scaling that are robust to local regularity properties. The findings have implications for large-scale convex optimization where global smoothness assumptions fail, while preserving rigorous convergence guarantees and explicit iteration complexity.
Abstract
This paper addresses the unconstrained minimization of smooth convex functions whose gradients are locally Holder continuous. Building on these results, we analyze the Scaled Gradient Algorithm (SGA) under local smoothness assumptions, proving its global convergence and iteration complexity. Furthermore, under local strong convexity and the Kurdyka-Lojasiewicz (KL) inequality, we establish linear convergence rates and provide explicit complexity bounds. In particular, we show that when the gradient is locally Lipschitz continuous, SGA attains linear convergence for any KL exponent. We then introduce and analyze an adaptive variant of SGA (AdaSGA), which automatically adjusts the scaling and step-size parameters. For this method, we show global convergence, and derive local linear rates under strong convexity.