Online Batch Selection for Faster Training of Neural Networks

TL;DR

The paper introduces online batch selection for stochastic gradient methods in deep learning, ranking datapoints by their latest loss and sampling batches with exponentially decaying probabilities to bias toward hard examples. This non-uniform batching is integrated with AdaDelta and Adam and evaluated on MNIST, where it yields about a fivefold speedup in training time while improving validation performance. The authors provide an analysis of computational overhead, explore hyperparameter optimization with CMA-ES, and extend the idea to related non-convex and CRF tasks, demonstrating both promise and limitations. Overall, the work suggests that simple, loss-rank-based batch selection can substantially accelerate training and invites further study on broader datasets and more sophisticated selection criteria.

Abstract

Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the dataset. While it is commonly accepted that batch size is an important parameter for offline tuning, the benefits of online selection of batches remain poorly understood. We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam. As the loss function to be minimized for the whole dataset is an aggregation of loss functions of individual datapoints, intuitively, datapoints with the greatest loss should be considered (selected in a batch) more frequently. However, the limitations of this intuition and the proper control of the selection pressure over time are open questions. We propose a simple strategy where all datapoints are ranked w.r.t. their latest known loss value and the probability to be selected decays exponentially as a function of rank. Our experimental results on the MNIST dataset suggest that selecting batches speeds up both AdaDelta and Adam by a factor of about 5.

Figures (8)

• Figure 1: Convergence curves of AdaDelta (Left Column) and Adam (Right Column) on MNIST dataset. The original algorithms are denoted by "default shuffle" (respectively, "default with random") when datapoints are shuffled and selected sequentially (respectively, uniformly at random). The value of $s$ denotes the ratio of probabilities to select the training datapoint with the greatest latest known loss rather than the smallest latest known loss. Legends with $s=s_{e_0} \rightarrow s_{e_{end}}$ correspond to an exponential change of $s$ from $s_{e_0}$ to $s_{e_{end}}$ as a function of epoch index $e$, see Eq. (\ref{['eq:ev1']}). Legends with the prefix "r" correspond to the case when the loss values for $r_{ratio} \cdot N=1.0N$ datapoints with the greatest latest known loss are recomputed $r_{freq}=0.5$ times per epoch. All curves are computed on the whole training set and smoothed by a moving average of the median of 11 runs.
• Figure 2: Minimum validation error found on MNIST for deep neural network trained with a given budget of time. The results correspond to the best hyperparameter settings found by CMA-ES after 1000 function evaluations. The original Adam and Adadelta algorithms are denoted by dotted lines, their versions with batch selection are denoted by solid lines.
• Figure 3: Convergence curves of AdaDelta (Left) and Adam (Right) on MNIST dataset. Bold lines correspond to our original approach, dotted lines correspond to its version with Importance Sampling. The value of $s$ denotes the ratio of probabilities to select the training datapoint with the greatest latest known loss rather than the smallest latest known loss. Legends with $s=s_{e_0} \rightarrow s_{e_{end}}$ correspond to an exponential change of $s$ from $s_{e_0}$ to $s_{e_{end}}$ as a function of epoch index $e$, see Eq. (\ref{['eq:ev1']}). Legends with the prefix "r" correspond to the case when the loss values for $r_{ratio} \cdot N=1.0N$ datapoints with the greatest latest known loss are recomputed $r_{freq}=0.5$ times per epoch. All curves are computed on the whole training set and smoothed by a moving average of the median of 11 runs.
• Figure 4: Objective minus optimal objective value against effective number of passes for different deterministic, stochastic, and semi-stochastic optimization strategies. Top-left: OCR, Top-right: CoNLL-2000, bottom-left: CoNLL-2002, bottom-right: POS-WSJ.
• Figure 5: Convergence curves of AdaDelta (Left Column) and Adam (Right Column) on MNIST dataset. The original algorithms are denoted by "default shuffle" (respectively, "default with random") when datapoints are shuffled and selected sequentially (respectively, uniformly at random). The value of $s$ denotes the ratio of probabilities to select the training datapoint with the greatest latest known loss rather than the smallest latest known loss. Legends with $s=s_{e_0} \rightarrow s_{e_{end}}$ correspond to an exponential change of $s$ from $s_{e_0}$ to $s_{e_{end}}$ as a function of epoch index $e$, see Eq. (\ref{['eq:ev1']}). Legends with the prefix "r" correspond to the case when the loss values for $r_{ratio} \cdot N=1.0N$ datapoints with the greatest latest known loss are recomputed $r_{freq}=0.5$ times per epoch. All curves are computed on the whole training set and smoothed by a moving average of the median of 11 runs.