A Flexible Algorithmic Framework for Strictly Convex Quadratic Minimization

TL;DR

The paper addresses the efficient minimization of strictly convex quadratic functions by introducing a Flexible Algorithmic Framework that forms a multi-directional search direction from a chosen set of sub-directions and selects step sizes to minimize the gradient norm in a weighted matrix norm. It proves linear convergence under relaxation and symmetric preconditioning and shows that many classical algorithms (e.g., Steepest Descent, Conjugate Gradients, Forsythe s-gradient methods) are special cases within this framework, while also proposing novel variants such as Gradient Descent with a Random Direction and momentum-augmented, random-direction methods. The key contributions include a unifying convergence theory, extensions to general norms, and practical prototype algorithms with numerical experiments illustrating trade-offs between iteration count and per-iteration cost. This framework provides a robust tool for rapid prototyping and rigorous analysis of gradient-based methods on quadratics, enabling principled development of new solvers with guaranteed linear convergence in broad settings.

Abstract

This paper presents an algorithmic framework for the minimization of strictly convex quadratic functions. The framework is flexible and generic. At every iteration the search direction is a linear combination of the negative gradient, as well as (possibly) several other `sub-search' directions, where the user determines which, and how many, sub-search directions to include. Then, a step size along each sub-direction is generated in such a way that the gradient is minimized (with respect to a matrix norm), over the hyperplane specified by the user chosen search directions. Theoretical machinery is developed, which shows that any algorithm that fits into the generic framework is guaranteed to converge at a linear rate. Moreover, these theoretical results hold even when relaxation and/or symmetric preconditioning is employed. Several state-of-the-art algorithms fit into this scheme, including steepest descent and conjugate gradients.

Figures (5)

• Figure 1: Forsythes $s-$gradient method with relaxation $\omega = 0.95$, compared with the unrelaxed methods. Function values are shown on the left, and the (squared) gradient norm is shown on the right.
• Figure 2: Conjugate direction methods, both with and without preconditioning. The function values are shown to take slightly fewer iterations when using a Jacobi preconditioner.
• Figure 3: A comparison of several relaxed $\ell$-MGD methods, with the Gradient Descent with a random direction method. The left plot shows the evolution of the function values, and the right plot shows the evolution of the squared 2-norm of the gradient as iterates progress.
• Figure 4: Results of perturbed gradient descent with $(\omega = 0.95)$ and without relaxation $(\omega = 1)$. Function values shown on the left and gradient norm is shown on the right.
• Figure 5: The CG and CR methods, compared with the 'enhanced' methods, which take $Ag_k$ or $r_k$ as an extra direction. The decrease in function values is shown on the left, and the gradient norm is shown on the right.

Theorems & Definitions (14)

• Definition 2
• Remark 5
• Lemma 6
• proof
• Remark 7
• Definition 8: Flexible Algorithmic Framework for \ref{['quadprob']}
• Lemma 9
• proof
• Theorem 10
• proof