The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization

TL;DR

The paper investigates early discarding in neural network hyperparameter optimization and finds that simple 1-Epoch baselines can be as effective or better than sophisticated methods like SHA, LCE, and PFN across multiple benchmarks. By evaluating under a fixed random-search stream and analyzing Pareto-fronts with hypervolume, the authors show that the i-Epoch approach (varying the fixed epoch budget) often dominates the trade-off between predictive performance and computational cost. A striking finding is the practical effectiveness of 1-Epoch, which remains competitive or superior in many tasks despite its simplicity, suggesting that learning curves are more favorable than previously assumed. The work argues for including 1-Epoch as a standard baseline in HPO studies and points toward future work on extending these ideas to wall-clock time and more complex architectures.

Abstract

To reach high performance with deep learning, hyperparameter optimization (HPO) is essential. This process is usually time-consuming due to costly evaluations of neural networks. Early discarding techniques limit the resources granted to unpromising candidates by observing the empirical learning curves and canceling neural network training as soon as the lack of competitiveness of a candidate becomes evident. Despite two decades of research, little is understood about the trade-off between the aggressiveness of discarding and the loss of predictive performance. Our paper studies this trade-off for several commonly used discarding techniques such as successive halving and learning curve extrapolation. Our surprising finding is that these commonly used techniques offer minimal to no added value compared to the simple strategy of discarding after a constant number of epochs of training. The chosen number of epochs depends mostly on the available compute budget. We call this approach i-Epoch (i being the constant number of epochs with which neural networks are trained) and suggest to assess the quality of early discarding techniques by comparing how their Pareto-Front (in consumed training epochs and predictive performance) complement the Pareto-Front of i-Epoch.

Figures (7)

• Figure 1: Hyperparameter optimization and its components including input/output, outer optimization loop exploring new hyperparameter configurations, inner optimization loop incrementally allocating training iterations ( what we study in this work) and selection of hyperparameters to return. In italic we specify the blocks to match with our experimental study.
• Figure 2: Comparing the any-time performance of various early discarding techniques during a random search (mean and one standard error over 10 repetitions) of 200 iterations (4 regression tasks). The two baseline strategies $1$-Epoch and $100$-Epoch method bound the trade-offs that can be achieved. The predictive performance of $1$-Epoch is at least of the same order of magnitude as other strategies while consuming a significantly smaller (the minimum in training epochs) number of training epochs.
• Figure 3: Comparing the any-time performance of various early discarding techniques during a random search (mean and one standard error over 10 repetitions) of 200 iterations (on 13 classification tasks).
• Figure 4: Multi-objective profiles built from spanning various levels of aggressiveness of early discarding methods (on 4 regression tasks). The estimated Pareto-Front of each method is shown in a plane line. The black dotted line corresponds to the estimated Pareto-Front including the methods altogether. It can be seen that the $i$-Epoch strategy spans more trade-offs (larger area) than other methods while never being significantly dominated.
• Figure 5: Multi-objective profiles built from spanning various levels of aggressiveness of early discarding methods (13 classification tasks). The estimated Pareto-Front of each method is shown in a plane line. The black dotted line corresponds to the estimated Pareto-Front including the methods altogether. It can be seen that the $i$-Epoch strategy spans more trade-offs (larger area) than other methods while never being significantly dominated.