Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions

TL;DR

A new spectral condition providing tighter upper bounds for problems with power law optimization trajectories is proposed, and first proofs of tight lower bounds for convergence rates of Steepest Descent and Conjugate Gradients under spectral power laws with general exponents are provided.

Abstract

Performance of optimization on quadratic problems sensitively depends on the low-lying part of the spectrum. For large (effectively infinite-dimensional) problems, this part of the spectrum can often be naturally represented or approximated by power law distributions, resulting in power law convergence rates for iterative solutions of these problems by gradient-based algorithms. In this paper, we propose a new spectral condition providing tighter upper bounds for problems with power law optimization trajectories. We use this condition to build a complete picture of upper and lower bounds for a wide range of optimization algorithms -- Gradient Descent, Steepest Descent, Heavy Ball, and Conjugate Gradients -- with an emphasis on the underlying schedules of learning rate and momentum. In particular, we demonstrate how an optimally accelerated method, its schedule, and convergence upper bound can be obtained in a unified manner for a given shape of the spectrum. Also, we provide first proofs of tight lower bounds for convergence rates of Steepest Descent and Conjugate Gradients under spectral power laws with general exponents. Our experiments show that the obtained convergence bounds and acceleration strategies are not only relevant for exactly quadratic optimization problems, but also fairly accurate when applied to the training of neural networks.

Figures (8)

• Figure 1: Spectral properties and GD loss of a MNIST classifier learned in a kernel regime. Left: The experimental trajectory $L(\mathbf w_n)$ of GD loss, the fitted power law \ref{['eq:lpropto']}, and the classical $O(n^{-1})$ bound \ref{['eq:gd_bound_polyak']}. Center: The eigenvalues $\lambda_k\propto k^{-\nu},\nu\approx1.37,$ and the cumulative distribution of target expansion coefficients, $\sum_{s=k}^{k_{\max}} \lambda_s c_s^2\propto k^{-\kappa}, \kappa\approx 0.34$. The target expansion coefficients $c_k$ are defined by the spectral expansion $\mathbf w_*=\sum_{k}c_k\mathbf e_k$ of the minimizer vector $\mathbf w_*$ over the eigenvectors $\mathbf e_k$. Right: The spectral measure $\rho([0,\lambda])=\sum_{k:\lambda_k<\lambda}\lambda_kc_k^2\propto \lambda^{\zeta}, \xi=\zeta=\tfrac{\kappa}{\nu}\approx 0.25$. See Section \ref{['sec:probdef']} for a general definition.
• Figure 2: Logical dependencies between sections with main results.
• Figure 3: The "flattened polynomial" $\overline{q}(x)$ associated with the worst-case spectral measure (see Theorem \ref{['ther:worst_case_loss']}). Left: The original polynomial $q(x)=p_n^2(x)$ and respective flattened polynomial $\overline{q}(x)$ for $p_n(x)$ associated with the Jacobi scheduled HB at step $n=7$ (see Section \ref{['sec:exact_power-law_measure']}). Right: Same as the left but zoomed in to a neighborhood of a single flat region of $\overline{q}(x)$. The orange parabola, placed at the respective root of $p_n(x)$ and normalized to match $q(x)$ at the right end of the flat region, is used to estimate the contribution of the flat region to the loss upper bound.
• Figure 4: SD applied to the uniform spectral distribution ($\rho((0,\lambda])=Q\lambda$) on $[0,1]$ converges to a period-2 oscillatory regime.
• Figure 5: Comparison of the experimental loss and different upper bounds for a kernel regression on the MNIST dataset. Left: Loss trajectories and respective bounds for constant learning rate GD (left subfigure) and Jacobi scheduled HB (right subfigure). For both GD and HB, the two upper bound curves are given by the functions $\widetilde{L}_n, \widetilde{L}'_n$ defined in Eqs. \ref{['eq:optimal_UB_our_condition']},\ref{['eq:optimal_UB_source_condition']}; in the GD case we additionally show the crude classical bound \ref{['eq:gd_bound_polyak']} corresponding to $\zeta=1$ and requiring a very large constant $C$. The colors of the dots reflect the optimized values of $\zeta,\zeta'$ in Eqs. \ref{['eq:optimal_UB_our_condition']}, \ref{['eq:optimal_UB_source_condition']}. Right: The actual spectral distribution $\rho$ and different spectral bounds $\rho((0,\lambda])\le Q\lambda^{\zeta}$ with varying $\zeta$ and respective optimal $Q.$

Theorems & Definitions (27)

• Lemma 2.1
• Theorem 4.1: see proof in Section \ref{['sec:proof_flattened']}
• Theorem 4.2
• Theorem 4.3
• Theorem 4.4
• Proposition 4.5
• Theorem 4.6
• Proposition 4.7
• Corollary 4.8
• Theorem 4.9