Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning complexity of gradient descent and conjugate gradient algorithms

Xianqi Jiao, Jia Liu, Zhiping Chen

TL;DR

The paper introduces a data-driven framework to quantify the learning complexity of tuning gradient-based optimization algorithms (GD and CG) via algorithm selection. A key contribution is a primal-dual integral-inspired cost that remains meaningful when iterations are halted and provides bounded pseudo-dimensions, enabling ERM-based guarantees. It establishes that GD admits a (C+ε,δ)-learnability with sample complexity m = ̃O(H^3/ε^2) under the new cost, and extends the approach to CG, yielding m = ̃O(H^4/ε^2). Together, these results demonstrate that optimal step-size and conjugate-parameter configurations can be identified with high probability from a finite sample, advancing data-driven algorithm design for large-scale optimization problems.

Abstract

Gradient Descent (GD) and Conjugate Gradient (CG) methods are among the most effective iterative algorithms for solving unconstrained optimization problems, particularly in machine learning and statistical modeling, where they are employed to minimize cost functions. In these algorithms, tunable parameters, such as step sizes or conjugate parameters, play a crucial role in determining key performance metrics, like runtime and solution quality. In this work, we introduce a framework that models algorithm selection as a statistical learning problem, and thus learning complexity can be estimated by the pseudo-dimension of the algorithm group. We first propose a new cost measure for unconstrained optimization algorithms, inspired by the concept of primal-dual integral in mixed-integer linear programming. Based on the new cost measure, we derive an improved upper bound for the pseudo-dimension of gradient descent algorithm group by discretizing the set of step size configurations. Moreover, we generalize our findings from gradient descent algorithm to the conjugate gradient algorithm group for the first time, and prove the existence a learning algorithm capable of probabilistically identifying the optimal algorithm with a sufficiently large sample size.

Learning complexity of gradient descent and conjugate gradient algorithms

TL;DR

The paper introduces a data-driven framework to quantify the learning complexity of tuning gradient-based optimization algorithms (GD and CG) via algorithm selection. A key contribution is a primal-dual integral-inspired cost that remains meaningful when iterations are halted and provides bounded pseudo-dimensions, enabling ERM-based guarantees. It establishes that GD admits a (C+ε,δ)-learnability with sample complexity m = ̃O(H^3/ε^2) under the new cost, and extends the approach to CG, yielding m = ̃O(H^4/ε^2). Together, these results demonstrate that optimal step-size and conjugate-parameter configurations can be identified with high probability from a finite sample, advancing data-driven algorithm design for large-scale optimization problems.

Abstract

Gradient Descent (GD) and Conjugate Gradient (CG) methods are among the most effective iterative algorithms for solving unconstrained optimization problems, particularly in machine learning and statistical modeling, where they are employed to minimize cost functions. In these algorithms, tunable parameters, such as step sizes or conjugate parameters, play a crucial role in determining key performance metrics, like runtime and solution quality. In this work, we introduce a framework that models algorithm selection as a statistical learning problem, and thus learning complexity can be estimated by the pseudo-dimension of the algorithm group. We first propose a new cost measure for unconstrained optimization algorithms, inspired by the concept of primal-dual integral in mixed-integer linear programming. Based on the new cost measure, we derive an improved upper bound for the pseudo-dimension of gradient descent algorithm group by discretizing the set of step size configurations. Moreover, we generalize our findings from gradient descent algorithm to the conjugate gradient algorithm group for the first time, and prove the existence a learning algorithm capable of probabilistically identifying the optimal algorithm with a sufficiently large sample size.

Paper Structure

This paper contains 21 sections, 21 theorems, 97 equations, 1 table.

Table of Contents

  1. Introduction
  2. Background of learning complexity
  3. Learning complexity of gradient descent algorithm
  4. Learning complexity on number of iterations for gradient descent algorithm
  5. A new cost function
  6. Learning complexity of gradient descent algorithm under the new cost function
  7. Learning complexity of conjugate gradient algorithm
  8. Conclusion
  9. Notation Table
  10. Details of Assumption 1.4
  11. Proof of Theorem 4
  12. Proof of Theorem 5
  13. Proof of Lemma 3
  14. Proof of Corollary 2
  15. Proof of Lemma 4
  16. ...and 6 more sections

Key Result

Theorem 1

Let $\mathcal{H}$ be a class of functions with domain $X$ and range in $[0,H]$, and suppose $\mathcal{H}$ has pseudo-dimension $d_\mathcal{H}$. For every distribution $\mathcal{D}$ over X, every $\epsilon>0$, and every $\delta \in (0,1]$, if for a suitable constant $k$ , then with probability at least $1-\delta$ over $m$ samples $x_1,x_2,\dots, x_m\sim \mathcal{D}$, for every $h \in \mathcal{H}$

Theorems & Definitions (36)

  • Definition 1: $(\epsilon,\delta)$-learning with probability k:94
  • Definition 2: Pseudo-dimension of the algorithm class p:84
  • Theorem 1: Uniform convergence a:99
  • Definition 3: Empirical risk minimization (ERM) v:91
  • Corollary 1
  • Theorem 2: Error estimation of cost function g:16
  • Lemma 1: Error estimation of iteration processes g:16
  • Theorem 3: Generalization Guarantees for GD on number of iterations g:16
  • Theorem 4: Error estimation of new cost function
  • Theorem 5: Generalization guarantees for GD on new cost function
  • ...and 26 more