Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Energy-Based Self-Adaptive Learning Rate for Stochastic Gradient Descent: Enhancing Unconstrained Optimization with VAV method

Jiahao Zhang, Christian Moya, Guang Lin

TL;DR

A novel energy-based self-adjustable learning rate optimization method designed for unconstrained optimization problems that incorporates an auxiliary variable to facilitate efficient energy approximation without backtracking while adhering to the unconditional energy dissipation law is introduced.

Abstract

Optimizing the learning rate remains a critical challenge in machine learning, essential for achieving model stability and efficient convergence. The Vector Auxiliary Variable (VAV) algorithm introduces a novel energy-based self-adjustable learning rate optimization method designed for unconstrained optimization problems. It incorporates an auxiliary variable $r$ to facilitate efficient energy approximation without backtracking while adhering to the unconditional energy dissipation law. Notably, VAV demonstrates superior stability with larger learning rates and achieves faster convergence in the early stage of the training process. Comparative analyses demonstrate that VAV outperforms Stochastic Gradient Descent (SGD) across various tasks. This paper also provides rigorous proof of the energy dissipation law and establishes the convergence of the algorithm under reasonable assumptions. Additionally, $r$ acts as an empirical lower bound of the training loss in practice, offering a novel scheduling approach that further enhances algorithm performance.

An Energy-Based Self-Adaptive Learning Rate for Stochastic Gradient Descent: Enhancing Unconstrained Optimization with VAV method

TL;DR

A novel energy-based self-adjustable learning rate optimization method designed for unconstrained optimization problems that incorporates an auxiliary variable to facilitate efficient energy approximation without backtracking while adhering to the unconditional energy dissipation law is introduced.

Abstract

Optimizing the learning rate remains a critical challenge in machine learning, essential for achieving model stability and efficient convergence. The Vector Auxiliary Variable (VAV) algorithm introduces a novel energy-based self-adjustable learning rate optimization method designed for unconstrained optimization problems. It incorporates an auxiliary variable to facilitate efficient energy approximation without backtracking while adhering to the unconditional energy dissipation law. Notably, VAV demonstrates superior stability with larger learning rates and achieves faster convergence in the early stage of the training process. Comparative analyses demonstrate that VAV outperforms Stochastic Gradient Descent (SGD) across various tasks. This paper also provides rigorous proof of the energy dissipation law and establishes the convergence of the algorithm under reasonable assumptions. Additionally, acts as an empirical lower bound of the training loss in practice, offering a novel scheduling approach that further enhances algorithm performance.

Paper Structure

This paper contains 29 sections, 6 theorems, 77 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Setting
  3. Previous Works
  4. Our Contribution
  5. VAV method
  6. Discussion of the Algorithm
  7. Baseline Algorithm
  8. Cost of the Proposed Method
  9. The Role of Parameters in the Proposed Algorithm
  10. Compatibility with other Optimization Techniques
  11. A Direct Application of $r:$ An energy-based self-adaptive scheduler
  12. Theoretical Analysis
  13. Numerical Experiments
  14. Toy Example: Rosenbrock function
  15. Regression Task
  16. ...and 14 more sections

Key Result

Theorem 4.1

The SAV algorithm is unconditionally energy dissipative, that is

Figures (4)

  • Figure 1: Training (left) and Testing Loss (right) of PINN on Burger's Equation.
  • Figure 2: Training (left) and Testing Loss (right) of PINN on Allen-Cahn Equation.
  • Figure 3: Training Loss per Epoch (Left), Test Accuracy (Center) and r$\_$square per Iteration (Right) on CIFAR10 dataset.
  • Figure 4: Training Loss per Epoch (Left), Test Accuracy (Center) and r$\_$square per Iteration (Right) on CIFAR100 dataset.

Theorems & Definitions (11)

  • Theorem 4.1
  • Theorem 4.2
  • Theorem 4.3
  • Theorem A.1
  • proof
  • Remark A.2
  • Theorem A.3
  • proof
  • Theorem A.4
  • proof
  • ...and 1 more