Eigenvector-based acceleration strategies for gradient-type methods

TL;DR

The paper tackles speeding gradient-type optimization on strictly convex quadratics and convex functions by relaxing the step length to break zigzag behavior and by exploiting spectral structure. It introduces eigenvector-based acceleration, Lanczos-based projection, and adaptive Lanczos strategies, then extends these ideas to nonquadratic convex problems with tolerant line searches to preserve convergence. Theoretical results show that residuals concentrate in a few extremal eigenmodes under relaxation, and practical schemes leverage this with Lanczos projections to substantially reduce iterations and gradient evaluations, demonstrated on large-scale problems and linked to AMG-like multigrid concepts. The work provides a cohesive framework and practical guidelines for integrating spectral accelerations into simple gradient-type methods for large-scale convex optimization.

Abstract

Several strategies are described and analyzed to speed-up gradient-type methods when applied to the minimization of strictly convex quadratics and strictly convex functions. The proposed techniques focus on relaxing the traditional optimal step length associated with gradient methods, including the steepest descent (SD) and the minimal residual (MR) methods. Such a relaxation avoids the well-known negative zigzag effect and allows the iterates to move in the entire space which in turn implies that every so often the search direction approaches some eigenvector of the underlying Hessian matrix. The proposed speedups then rely on taking advantage of the properties of the Lanczos method once a search direction that approaches an eigenvector has been identified in order to accelerate the convergence towards the global minimizer. After analyzing the proposed strategies, we illustrate them on the global minimization of strictly convex functions.

Figures (14)

• Figure 1: Sequence of Rayleigh quotients (${1/\alpha_k }$) with random initial gradient, $\sigma = 0.9$ and $\sigma = 1.8$, for both the MR (left) and SD (right) algorithms. The top and bottom lines are the largest and smallest eigenvalues. The dashed black line is $\sigma(\lambda_1 + \lambda_n)/2$ while the dashed red line is $(\lambda_1 + \lambda_n)/2$.
• Figure 2: Components of the normalized residuals in the eigenbasis for iterations $k=3,7,11,33$ with the MR (left) and SD (right) algorithms using $\sigma = 0.9$ or $\sigma=1.8$.
• Figure 3: Convergence of $|\beta_{i,k}|$ for different values of $i$ and $\sigma=0.8$ in the case of the MR (left) or SD (right) algorithms. We used two different random initial guesses. The normalized residuals are supported by a few extremal modes (here, mostly the lowest mode $i=1$ and a few of the highest modes between $i=n$ and $i=n-5$), and all the intermediate modes vanish asymptotically. Only a few modes are plotted for the sake of clarity.
• Figure 4: Eigenvector based acceleration ($\sigma=0.8$, $\epsilon_{\rm eig} = 0.8$): when the residual is close to an eigenvector (that is, the yellow line goes below the threshold), the acceleration is activated (a drop is observed on the red line). On the left, the blue line is for comparison with MR ($\tau_k=1$ at every iteration): it converges much slower due to the zigzag effect. On the right, the blue line is MR with $\tau_k=\sigma\in(0,1)$ at every iteration: it behaves similarly to the eigenvector acceleration.
• Figure 5: Lanczos based acceleration ($\sigma=0.8$, $\epsilon_{\rm eig} = 0.8$): using a few steps ($m=5$) of the Lanczos algorithm makes the acceleration process more efficient, both in terms of iterations (left) and number of matrix-vector products (right). Note that, for the Lanczos-based acceleration, the number of matrix-vector products is higher than the number of iterations because every time a Lanczos projection is performed, additional matrix-vector products are used.

Theorems & Definitions (15)

• Lemma 3.1
• Theorem 3.2
• Proposition 3.3
• Lemma 3.4
• Proposition 3.5
• Corollary 3.6
• Remark 3.7
• Remark 3.8: Case $m=1$
• Corollary 3.9
• Remark 3.10: Givens rotations