Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The duality structure gradient descent algorithm: analysis and applications to neural networks

Thomas Flynn

TL;DR

The paper tackles non-convex neural-network optimization where the gradient is not globally Lipschitz by introducing Duality Structure Gradient Descent (DSGD), a layer-wise greedy method that updates one layer per iteration using a geometry induced by a family of norms tied to the network structure. It jointly develops a generalized smoothness framework and a duality-map based update rule, enabling non-asymptotic convergence results in both deterministic and stochastic settings. The main theoretical contribution is a bound on the expected time to reach an approximate stationary point, plus corollaries that recover classical SGD results under special norm choices. Empirically, DSGD is demonstrated on standard benchmarks, illustrating how layer-wise updates and non-Euclidean geometry influence update distribution and training dynamics.

Abstract

The training of machine learning models is typically carried out using some form of gradient descent, often with great success. However, non-asymptotic analyses of first-order optimization algorithms typically employ a gradient smoothness assumption (formally, Lipschitz continuity of the gradient) that is too strong to be applicable in the case of deep neural networks. To address this, we propose an algorithm named duality structure gradient descent (DSGD) that is amenable to non-asymptotic performance analysis, under mild assumptions on the training set and network architecture. The algorithm can be viewed as a form of layer-wise coordinate descent, where at each iteration the algorithm chooses one layer of the network to update. The decision of what layer to update is done in a greedy fashion, based on a rigorous lower bound on the improvement of the objective function for each choice of layer. In the analysis, we bound the time required to reach approximate stationary points, in both the deterministic and stochastic settings. The convergence is measured in terms of a parameter-dependent family of norms that is derived from the network architecture and designed to confirm a smoothness-like property on the gradient of the training loss function. We empirically demonstrate the behavior of DSGD in several neural network training scenarios.

The duality structure gradient descent algorithm: analysis and applications to neural networks

TL;DR

The paper tackles non-convex neural-network optimization where the gradient is not globally Lipschitz by introducing Duality Structure Gradient Descent (DSGD), a layer-wise greedy method that updates one layer per iteration using a geometry induced by a family of norms tied to the network structure. It jointly develops a generalized smoothness framework and a duality-map based update rule, enabling non-asymptotic convergence results in both deterministic and stochastic settings. The main theoretical contribution is a bound on the expected time to reach an approximate stationary point, plus corollaries that recover classical SGD results under special norm choices. Empirically, DSGD is demonstrated on standard benchmarks, illustrating how layer-wise updates and non-Euclidean geometry influence update distribution and training dynamics.

Abstract

The training of machine learning models is typically carried out using some form of gradient descent, often with great success. However, non-asymptotic analyses of first-order optimization algorithms typically employ a gradient smoothness assumption (formally, Lipschitz continuity of the gradient) that is too strong to be applicable in the case of deep neural networks. To address this, we propose an algorithm named duality structure gradient descent (DSGD) that is amenable to non-asymptotic performance analysis, under mild assumptions on the training set and network architecture. The algorithm can be viewed as a form of layer-wise coordinate descent, where at each iteration the algorithm chooses one layer of the network to update. The decision of what layer to update is done in a greedy fashion, based on a rigorous lower bound on the improvement of the objective function for each choice of layer. In the analysis, we bound the time required to reach approximate stationary points, in both the deterministic and stochastic settings. The convergence is measured in terms of a parameter-dependent family of norms that is derived from the network architecture and designed to confirm a smoothness-like property on the gradient of the training loss function. We empirically demonstrate the behavior of DSGD in several neural network training scenarios.

Paper Structure

This paper contains 9 sections, 1 theorem, 4 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Layer-wise Lipschitz property
  3. Duality Structure Gradient Descent
  4. Intralayer update rules
  5. Outline
  6. Related work
  7. Notation
  8. Motivating example
  9. Duality Structure Gradient Descent

Key Result

Proposition 2.1

The function $f$ defined in Equation opt-obj has unbounded second derivatives: $\sup_{w\in\mathbb{R}^4}\|\tfrac{\partial^2 f}{\partial w^{2}}(w)\| = \infty.$

Figures (1)

  • Figure 1: The small network used as a motivating example in Section \ref{['funsect']}. We show that the training problem of mapping the input $1$ to the output $0$, using the logistic activation function and squared-error loss, leads to an objective where the gradient is not Lipschitz continuous.

Theorems & Definitions (3)

  • Proposition 2.1
  • definition thmcounterdefinition
  • definition thmcounterdefinition