Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

New gradient methods with 3 dimensional quadratic termination

Yixin Xie, Jin-Peng Liu, Cong Sun, Ya-Xiang Yuan

TL;DR

This paper proposes a novel NY stepsize that achieves 5-step termination for 3D quadratic minimization and embeds it in a cyclic gradient framework to form a new gradient method. It then extends to general unconstrained optimization by approximating Cauchy steps with quadratic interpolation and employing an improved GLL line search, proving global convergence and establishing sublinear and R-linear convergence under appropriate conditions. Numerical experiments show the proposed NY/ANY methods outperform state-of-the-art gradient-based methods in CPU time and line-search efficiency, including high-dimensional and ill-conditioned problems. The approach offers a practical, low-cost alternative for large-scale optimization with strong termination guarantees in low-dimensional subspaces and robust performance in broader settings.

Abstract

A new stepsize for gradient method is proposed. Combining it with the exact line search stepsizes, the gradient method achieves the optimal solution in 5 steps for 3 dimensional quadratic function minimization problem. The new stepsize is plugged in the cyclic stepsize update strategy, and a new gradient method is proposed. By applying the quadratic interpolation for Cauchy approximation, the proposed gradient method is extended to solve general unconstrained problem. With the improved GLL line search, the global convergence of the proposed method is proved. Furthermore, its sublinear convergence rate for convex problems and R-linear convergence rate for problems with quadratic functional growth property are analyzed. Numerical results show that our proposed algorithm enjoys good performances in terms of computational cost, and line search requires very few trial stepsizes.

New gradient methods with 3 dimensional quadratic termination

TL;DR

This paper proposes a novel NY stepsize that achieves 5-step termination for 3D quadratic minimization and embeds it in a cyclic gradient framework to form a new gradient method. It then extends to general unconstrained optimization by approximating Cauchy steps with quadratic interpolation and employing an improved GLL line search, proving global convergence and establishing sublinear and R-linear convergence under appropriate conditions. Numerical experiments show the proposed NY/ANY methods outperform state-of-the-art gradient-based methods in CPU time and line-search efficiency, including high-dimensional and ill-conditioned problems. The approach offers a practical, low-cost alternative for large-scale optimization with strong termination guarantees in low-dimensional subspaces and robust performance in broader settings.

Abstract

A new stepsize for gradient method is proposed. Combining it with the exact line search stepsizes, the gradient method achieves the optimal solution in 5 steps for 3 dimensional quadratic function minimization problem. The new stepsize is plugged in the cyclic stepsize update strategy, and a new gradient method is proposed. By applying the quadratic interpolation for Cauchy approximation, the proposed gradient method is extended to solve general unconstrained problem. With the improved GLL line search, the global convergence of the proposed method is proved. Furthermore, its sublinear convergence rate for convex problems and R-linear convergence rate for problems with quadratic functional growth property are analyzed. Numerical results show that our proposed algorithm enjoys good performances in terms of computational cost, and line search requires very few trial stepsizes.
Paper Structure (11 sections, 7 theorems, 66 equations, 5 figures, 2 tables, 3 algorithms)

This paper contains 11 sections, 7 theorems, 66 equations, 5 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. New Stepsize with $3$ Dimensional $5$-step Termination
  3. New Gradient Method
  4. Quadratic case
  5. General case
  6. Numerical Tests
  7. Algorithm comparison
  8. Line search analysis
  9. High dimensional tests
  10. Conclusion
  11. The Improved GLL Line Search

Key Result

Theorem 2.1

The stepsizes $\alpha^{NY(i)}$ ($i=1,2,3$) defined in $(NYsteps)$ satisfy the following order:

Figures (5)

  • Figure 4.1: Performance profile on the CPU time metric
  • Figure 4.2: Performance profile on the iteration metric
  • Figure 4.3: Performance profile on the number of function value metric
  • Figure 4.4: Performance profile on the number of gradient value metric
  • Figure 4.5: Comparison of average line search counts per iteration.

Theorems & Definitions (8)

  • Theorem 2.1
  • Theorem 2.2
  • Theorem 3.1
  • Lemma 3.1
  • Theorem 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Remark 4.1