Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

General Cyclical Training of Neural Networks

Leslie N. Smith

TL;DR

The paper addresses the rigidity of fixed training settings by proposing general cyclical training, a framework where early epochs are easy, middle epochs explore the full problem space with harder conditions, and final epochs fine-tune on confident samples. It formalizes this via a schedule over training settings and a cyclical factor $f_c$ that shapes cycles, and demonstrates multiple manifestations across hyper-parameters, data, losses, and models. Key contributions include cyclical weight decay, cyclical batch size, cyclical focal loss, cyclical softmax temperature, cyclical data augmentation, cyclical gradient clipping, and cyclical semi-supervised learning, with empirical gains on CIFAR and ImageNet and practical guidance for pre-training and knowledge distillation. The work offers a unifying perspective that can reduce hyperparameter tuning and improve generalization, supplemented by reproducible code to enable adoption and further research.

Abstract

This paper describes the principle of "General Cyclical Training" in machine learning, where training starts and ends with "easy training" and the "hard training" happens during the middle epochs. We propose several manifestations for training neural networks, including algorithmic examples (via hyper-parameters and loss functions), data-based examples, and model-based examples. Specifically, we introduce several novel techniques: cyclical weight decay, cyclical batch size, cyclical focal loss, cyclical softmax temperature, cyclical data augmentation, cyclical gradient clipping, and cyclical semi-supervised learning. In addition, we demonstrate that cyclical weight decay, cyclical softmax temperature, and cyclical gradient clipping (as three examples of this principle) are beneficial in the test accuracy performance of a trained model. Furthermore, we discuss model-based examples (such as pretraining and knowledge distillation) from the perspective of general cyclical training and recommend some changes to the typical training methodology. In summary, this paper defines the general cyclical training concept and discusses several specific ways in which this concept can be applied to training neural networks. In the spirit of reproducibility, the code used in our experiments is available at \url{https://github.com/lnsmith54/CFL}.

General Cyclical Training of Neural Networks

TL;DR

The paper addresses the rigidity of fixed training settings by proposing general cyclical training, a framework where early epochs are easy, middle epochs explore the full problem space with harder conditions, and final epochs fine-tune on confident samples. It formalizes this via a schedule over training settings and a cyclical factor that shapes cycles, and demonstrates multiple manifestations across hyper-parameters, data, losses, and models. Key contributions include cyclical weight decay, cyclical batch size, cyclical focal loss, cyclical softmax temperature, cyclical data augmentation, cyclical gradient clipping, and cyclical semi-supervised learning, with empirical gains on CIFAR and ImageNet and practical guidance for pre-training and knowledge distillation. The work offers a unifying perspective that can reduce hyperparameter tuning and improve generalization, supplemented by reproducible code to enable adoption and further research.

Abstract

This paper describes the principle of "General Cyclical Training" in machine learning, where training starts and ends with "easy training" and the "hard training" happens during the middle epochs. We propose several manifestations for training neural networks, including algorithmic examples (via hyper-parameters and loss functions), data-based examples, and model-based examples. Specifically, we introduce several novel techniques: cyclical weight decay, cyclical batch size, cyclical focal loss, cyclical softmax temperature, cyclical data augmentation, cyclical gradient clipping, and cyclical semi-supervised learning. In addition, we demonstrate that cyclical weight decay, cyclical softmax temperature, and cyclical gradient clipping (as three examples of this principle) are beneficial in the test accuracy performance of a trained model. Furthermore, we discuss model-based examples (such as pretraining and knowledge distillation) from the perspective of general cyclical training and recommend some changes to the typical training methodology. In summary, this paper defines the general cyclical training concept and discusses several specific ways in which this concept can be applied to training neural networks. In the spirit of reproducibility, the code used in our experiments is available at \url{https://github.com/lnsmith54/CFL}.
Paper Structure (15 sections, 7 equations, 3 figures, 4 tables)

This paper contains 15 sections, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. General Cyclical Training
  3. Algorithmic Examples
  4. Hyper-parameters and Regularization
  5. Loss Functions
  6. Data-Based Examples
  7. Data Augmentation
  8. Data
  9. Semi-supervised Learning
  10. Model-Based Examples
  11. Transfer Learning and Pre-training
  12. Knowledge Distillation
  13. Discussion
  14. Software and Implementation
  15. Command Lines and Hyper-parameters

Figures (3)

  • Figure 1: Effect of $f_c$ on the cyclical schedule: This figure shows the impact of different values for $f_c$ on the cyclical schedule.
  • Figure 2: CWD on ImageNet: This figure shows the ImageNet test accuracy curve during training for cyclical weight decay versus a constant weight decay. The difference between the curves is minor, but shows cyclical weight decay allows a slightly faster rise in test accuracy.
  • Figure 3: Sensitivity of $f_c$: This figure shows the ImageNet test accuracy for a range of values for $f_c$ when trained with cyclical weight decay. The accuracy changes are small over this range of $f_c$ values and $f_c = 2$ or 4 are generally good values.