Randomness Helps Rigor: A Probabilistic Learning Rate Scheduler Bridging Theory and Deep Learning Practice
Dahlia Devapriya, Thulasi Tholeti, Janani Suresh, Sheetal Kalyani
TL;DR
This work introduces a Probabilistic Learning Rate Scheduler (PLRS) that samples the learning rate from a uniform interval, yielding η_{t+1}=η_c+u_{t+1} with u_{t+1}∼U[L_{min}-η_c,L_{max}-η_c], and demonstrates SGD-PLRS converges to a ν-SOSP under standard smoothness and strict-saddle assumptions, addressing the lack of theoretical guarantees for non-monotone LR schedulers. It shows that the LR randomness acts as multiplicative noise, enabling saddle-point escape and improved convergence properties without requiring knowledge of problem-specific constants. The paper provides rigorous convergence proofs and extensive empirical validation across ResNet, WRN, VGG, and DenseNet on CIFAR-10/100 and Tiny ImageNet, where PLRS matches or surpasses cosine, knee, one-cycle, and multi-step schedulers and often yields faster, more stable convergence. By linking cyclical/deterministic schedulers under a probabilistic framework, PLRS offers a practical, theory-backed approach that can reduce training time and resource usage while maintaining or improving accuracy.
Abstract
Learning rate schedulers have shown great success in speeding up the convergence of learning algorithms in practice. However, their convergence to a minimum has not been proven theoretically. This difficulty mainly arises from the fact that, while traditional convergence analysis prescribes to monotonically decreasing (or constant) learning rates, schedulers opt for rates that often increase and decrease through the training epochs. In this work, we aim to bridge the gap by proposing a probabilistic learning rate scheduler (PLRS) that does not conform to the monotonically decreasing condition, with provable convergence guarantees. To cement the relevance and utility of our work in modern day applications, we show experimental results on deep neural network architectures such as ResNet, WRN, VGG, and DenseNet on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. We show that PLRS performs as well as or better than existing state-of-the-art learning rate schedulers in terms of convergence as well as accuracy. For example, while training ResNet-110 on the CIFAR-100 dataset, we outperform the state-of-the-art knee scheduler by $1.56\%$ in terms of classification accuracy. Furthermore, on the Tiny ImageNet dataset using ResNet-50 architecture, we show a significantly more stable convergence than the cosine scheduler and a better classification accuracy than the existing schedulers.