Randomness and Interpolation Improve Gradient Descent

TL;DR

This paper addresses optimization in SGD-based training by introducing two methods: Noise-Regularized Stochastic Gradient Descent (NRSGD) and Interpolational Accelerating Gradient Descent (IAGD). NRSGD injects gradient noise with $n \sim N(\bar{\nabla}, \sigma_{\nabla})$ to regularize updates, resulting in $x_{i+1}=x_i - \eta_i ( w(\nabla - n) + n )$ and robustness to numerical errors. IAGD leverages second-order Newton interpolation to predict the next gradient and updates via $x_{i+1}=x_i - \eta_i\nabla_i - \eta_{i-1}f(\nabla_i)$, reducing the effective iterations needed. Empirical results on CIFAR-10/100 with AlexNet and LeNet-5 show IAGD achieves fast convergence while NRSGD improves test accuracy and robustness; the work indicates these strategies can enhance SGD and potentially generalize to other optimizers.

Abstract

Based on Stochastic Gradient Descent (SGD), the paper introduces two optimizers, named Interpolational Accelerating Gradient Descent (IAGD) as well as Noise-Regularized Stochastic Gradient Descent (NRSGD). IAGD leverages second-order Newton Interpolation to expedite the convergence process during training, assuming relevancy in gradients between iterations. To avoid over-fitting, NRSGD incorporates a noise regularization technique that introduces controlled noise to the gradients during the optimization process. Comparative experiments of this research are conducted on the CIFAR-10, and CIFAR-100 datasets, benchmarking different CNNs(Convolutional Neural Networks) with IAGD and NRSGD against classical optimizers in Keras Package. Results demonstrate the potential of those two viable improvement methods in SGD, implicating the effectiveness of the advancements.

Figures (5)

• Figure 1: Except Convergence Process
• Figure 2: AlexNet Framework b8
• Figure 3: LeNet-5 Framework b10
• Figure 4: Performance of LeNet-5 with Tested Optimizers
• Figure 5: Performance of AlexNet with Tested Optimizers