Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Rate Curriculum

Florinel-Alin Croitoru, Nicolae-Catalin Ristea, Radu Tudor Ionescu, Nicu Sebe

TL;DR

This work proposes Learning Rate Curriculum (LeRaC), a model-level curriculum that assigns per-layer learning rates to create an easy-to-hard training progression without sorting data samples. By keeping higher rates in shallower layers and gradually equalizing rates across layers by epoch $k$ using an exponential scheduler with base $c=10$, LeRaC mitigates early-layer noise and stabilizes optimization. Across 12 diverse datasets spanning vision, language, and audio, and multiple architectures (CNNs, RNNs, transformers), LeRaC yields consistent improvements over conventional training and outperforms Curriculum by Smoothing (CBS) in most scenarios, without adding training time. The findings support LeRaC as a broadly applicable, efficient curriculum strategy with strong empirical gains and clear theoretical motivation based on layer-wise signal-to-noise considerations.

Abstract

Most curriculum learning methods require an approach to sort the data samples by difficulty, which is often cumbersome to perform. In this work, we propose a novel curriculum learning approach termed Learning Rate Curriculum (LeRaC), which leverages the use of a different learning rate for each layer of a neural network to create a data-agnostic curriculum during the initial training epochs. More specifically, LeRaC assigns higher learning rates to neural layers closer to the input, gradually decreasing the learning rates as the layers are placed farther away from the input. The learning rates increase at various paces during the first training iterations, until they all reach the same value. From this point on, the neural model is trained as usual. This creates a model-level curriculum learning strategy that does not require sorting the examples by difficulty and is compatible with any neural network, generating higher performance levels regardless of the architecture. We conduct comprehensive experiments on 12 data sets from the computer vision (CIFAR-10, CIFAR-100, Tiny ImageNet, ImageNet-200, Food-101, UTKFace, PASCAL VOC), language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D) domains, considering various convolutional (ResNet-18, Wide-ResNet-50, DenseNet-121, YOLOv5), recurrent (LSTM) and transformer (CvT, BERT, SepTr) architectures. We compare our approach with the conventional training regime, as well as with Curriculum by Smoothing (CBS), a state-of-the-art data-agnostic curriculum learning approach. Unlike CBS, our performance improvements over the standard training regime are consistent across all data sets and models. Furthermore, we significantly surpass CBS in terms of training time (there is no additional cost over the standard training regime for LeRaC). Our code is freely available at: https://github.com/CroitoruAlin/LeRaC.

Learning Rate Curriculum

TL;DR

This work proposes Learning Rate Curriculum (LeRaC), a model-level curriculum that assigns per-layer learning rates to create an easy-to-hard training progression without sorting data samples. By keeping higher rates in shallower layers and gradually equalizing rates across layers by epoch using an exponential scheduler with base , LeRaC mitigates early-layer noise and stabilizes optimization. Across 12 diverse datasets spanning vision, language, and audio, and multiple architectures (CNNs, RNNs, transformers), LeRaC yields consistent improvements over conventional training and outperforms Curriculum by Smoothing (CBS) in most scenarios, without adding training time. The findings support LeRaC as a broadly applicable, efficient curriculum strategy with strong empirical gains and clear theoretical motivation based on layer-wise signal-to-noise considerations.

Abstract

Most curriculum learning methods require an approach to sort the data samples by difficulty, which is often cumbersome to perform. In this work, we propose a novel curriculum learning approach termed Learning Rate Curriculum (LeRaC), which leverages the use of a different learning rate for each layer of a neural network to create a data-agnostic curriculum during the initial training epochs. More specifically, LeRaC assigns higher learning rates to neural layers closer to the input, gradually decreasing the learning rates as the layers are placed farther away from the input. The learning rates increase at various paces during the first training iterations, until they all reach the same value. From this point on, the neural model is trained as usual. This creates a model-level curriculum learning strategy that does not require sorting the examples by difficulty and is compatible with any neural network, generating higher performance levels regardless of the architecture. We conduct comprehensive experiments on 12 data sets from the computer vision (CIFAR-10, CIFAR-100, Tiny ImageNet, ImageNet-200, Food-101, UTKFace, PASCAL VOC), language (BoolQ, QNLI, RTE) and audio (ESC-50, CREMA-D) domains, considering various convolutional (ResNet-18, Wide-ResNet-50, DenseNet-121, YOLOv5), recurrent (LSTM) and transformer (CvT, BERT, SepTr) architectures. We compare our approach with the conventional training regime, as well as with Curriculum by Smoothing (CBS), a state-of-the-art data-agnostic curriculum learning approach. Unlike CBS, our performance improvements over the standard training regime are consistent across all data sets and models. Furthermore, we significantly surpass CBS in terms of training time (there is no additional cost over the standard training regime for LeRaC). Our code is freely available at: https://github.com/CroitoruAlin/LeRaC.
Paper Structure (19 sections, 3 theorems, 32 equations, 10 figures, 18 tables)

This paper contains 19 sections, 3 theorems, 32 equations, 10 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Curriculum Learning
  4. Learning Rate Schedulers
  5. Optimizers
  6. Method
  7. Experiments
  8. Data Sets
  9. Experimental Setup
  10. Domain-Specific Preprocessing
  11. Preliminary Results
  12. Main Results
  13. More Comparative Results
  14. Ablation Studies
  15. Discussion
  16. ...and 4 more sections

Key Result

Theorem 1

Let $s_1=u_1+z_1$ and $s_2=u_2+z_2$ be two signals, where $u_1$ and $u_2$ are the clean components, and $z_1$ and $z_2$ are the noise components. The signal-to-noise ratio of the product between the two signals is lower than the signal-to-noise ratios of the two signals, i.e.:

Figures (10)

  • Figure 1: Training based on Learning Rate Curriculum.
  • Figure 2: Convolving an image of a car with random noise filters progressively increases the level of noise in the features. A theoretical proof of this observation is given in Appendix \ref{['app_proof']}.
  • Figure 3: Validation accuracy (on the y-axis) versus training time (on the x-axis) for four distinct architectures. The number of training epochs is the same for both LeRaC and CBS, the observable time difference being caused by the overhead of CBS due to the Gaussian kernel layers.
  • Figure 4: Average SNR of the feature maps at each layer of the randomly initialized LeNet architecture. The SNR at each layer is averaged for 100 randomly picked images from the CIFAR-100 data set. For the later layers, the SNR is negative because the signal is dominated by noise.
  • Figure 5: Test accuracy (on the y-axis) versus training time (on the x-axis) for ResNet-18 on CIFAR-100 with various curriculum schedulers. The dashed line corresponds to the conventional regime, while the continuous lines correspond to LeRaC with various schedulers. Best viewed in color.
  • ...and 5 more figures

Theorems & Definitions (8)

  • Theorem 1
  • proof
  • Definition 1
  • Definition 2
  • Theorem A.1
  • proof
  • Corollary 1
  • proof