Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cyclical Log Annealing as a Learning Rate Scheduler

Philip Naveen

TL;DR

This paper addresses learning rate scheduling in gradient-based optimization and introduces cyclical log annealing, a restart-based method that adjusts the learning rate $\eta_t$ using a logarithmic pattern during training in the online convex optimization setting. The goal is to quickly escape poor local minima while preserving stable convergence, offering an alternative to cosine annealing and step-based schedules. Empirical results on CIFAR-10 show that log annealing achieves performance comparable to cosine annealing on ResNet34 and transformer-enhanced architectures, with sometimes better late-stage convergence in some models. The authors discuss potential benefits for greedy solvers and future extension to GANs, while acknowledging the need for more extensive hyperparameter studies.

Abstract

A learning rate scheduler is a predefined set of instructions for varying search stepsizes during model training processes. This paper introduces a new logarithmic method using harsh restarting of step sizes through stochastic gradient descent. Cyclical log annealing implements the restart pattern more aggressively to maybe allow the usage of more greedy algorithms on the online convex optimization framework. The algorithm was tested on the CIFAR-10 image datasets, and seemed to perform analogously with cosine annealing on large transformer-enhanced residual neural networks. Future experiments would involve testing the scheduler in generative adversarial networks and finding the best parameters for the scheduler with more experiments.

Cyclical Log Annealing as a Learning Rate Scheduler

TL;DR

This paper addresses learning rate scheduling in gradient-based optimization and introduces cyclical log annealing, a restart-based method that adjusts the learning rate using a logarithmic pattern during training in the online convex optimization setting. The goal is to quickly escape poor local minima while preserving stable convergence, offering an alternative to cosine annealing and step-based schedules. Empirical results on CIFAR-10 show that log annealing achieves performance comparable to cosine annealing on ResNet34 and transformer-enhanced architectures, with sometimes better late-stage convergence in some models. The authors discuss potential benefits for greedy solvers and future extension to GANs, while acknowledging the need for more extensive hyperparameter studies.

Abstract

A learning rate scheduler is a predefined set of instructions for varying search stepsizes during model training processes. This paper introduces a new logarithmic method using harsh restarting of step sizes through stochastic gradient descent. Cyclical log annealing implements the restart pattern more aggressively to maybe allow the usage of more greedy algorithms on the online convex optimization framework. The algorithm was tested on the CIFAR-10 image datasets, and seemed to perform analogously with cosine annealing on large transformer-enhanced residual neural networks. Future experiments would involve testing the scheduler in generative adversarial networks and finding the best parameters for the scheduler with more experiments.
Paper Structure (9 sections, 6 equations, 3 figures)

This paper contains 9 sections, 6 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Gradient Based Optimization
  3. Restarting Mechan
  4. Experiments
  5. CIFAR-10 Classification
  6. Transformer CIFAR-10 Classification
  7. Discussion
  8. Conclusions
  9. acknowledgements

Figures (3)

  • Figure 1: Learning Rate Schedulers; shared parameters are initial decay epochs=1, min decay lr=0.001, restart interval=1, restart interval multiplier=1.5, restart lr=0.05, warmup epochs=1, warmup start lr=0.0001 for annealing schedulers.
  • Figure 2: The Effect of Learning Rate Schedulers on CIFAR-10 Image Classification using ResNet34
  • Figure 3: The Effect of Learning Rate Schedulers on CIFAR-10 Image Classification using Transformers