Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization

Thomas Nagler, Lennart Schneider, Bernd Bischl, Matthias Feurer

TL;DR

It is shown that reshuffling the splits for every configuration often improves the final model's generalization performance on unseen data, and a bound on the expected regret in the limiting regime is provided.

Abstract

Hyperparameter optimization is crucial for obtaining peak performance of machine learning models. The standard protocol evaluates various hyperparameter configurations using a resampling estimate of the generalization error to guide optimization and select a final hyperparameter configuration. Without much evidence, paired resampling splits, i.e., either a fixed train-validation split or a fixed cross-validation scheme, are often recommended. We show that, surprisingly, reshuffling the splits for every configuration often improves the final model's generalization performance on unseen data. Our theoretical analysis explains how reshuffling affects the asymptotic behavior of the validation loss surface and provides a bound on the expected regret in the limiting regime. This bound connects the potential benefits of reshuffling to the signal and noise characteristics of the underlying optimization problem. We confirm our theoretical results in a controlled simulation study and demonstrate the practical usefulness of reshuffling in a large-scale, realistic hyperparameter optimization experiment. While reshuffling leads to test performances that are competitive with using fixed splits, it drastically improves results for a single train-validation holdout protocol and can often make holdout become competitive with standard CV while being computationally cheaper.

Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization

TL;DR

It is shown that reshuffling the splits for every configuration often improves the final model's generalization performance on unseen data, and a bound on the expected regret in the limiting regime is provided.

Abstract

Hyperparameter optimization is crucial for obtaining peak performance of machine learning models. The standard protocol evaluates various hyperparameter configurations using a resampling estimate of the generalization error to guide optimization and select a final hyperparameter configuration. Without much evidence, paired resampling splits, i.e., either a fixed train-validation split or a fixed cross-validation scheme, are often recommended. We show that, surprisingly, reshuffling the splits for every configuration often improves the final model's generalization performance on unseen data. Our theoretical analysis explains how reshuffling affects the asymptotic behavior of the validation loss surface and provides a bound on the expected regret in the limiting regime. This bound connects the potential benefits of reshuffling to the signal and noise characteristics of the underlying optimization problem. We confirm our theoretical results in a controlled simulation study and demonstrate the practical usefulness of reshuffling in a large-scale, realistic hyperparameter optimization experiment. While reshuffling leads to test performances that are competitive with using fixed splits, it drastically improves results for a single train-validation holdout protocol and can often make holdout become competitive with standard CV while being computationally cheaper.
Paper Structure (35 sections, 4 theorems, 61 equations, 30 figures, 5 tables)

This paper contains 35 sections, 4 theorems, 61 equations, 30 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Theoretical Analysis
  3. Problem Statement and Setup
  4. How Reshuffling Affects the Loss Surface
  5. How Reshuffling Affects HPO Performance
  6. Simulation Study
  7. Design
  8. Results
  9. Benchmark Experiments
  10. Experimental Setup
  11. Experimental Results
  12. Results of Reshuffling Different Resamplings
  13. Results for BO and Reshuffling
  14. Discussion
  15. Limitations
  16. ...and 20 more sections

Key Result

Theorem 2.1

Under regularity conditions stated in Appendix app:proof-normality, it holds where and where the expectation is taken over a training set $\mathcal{T}$ of size $n$ and two fresh samples $\bm{Z}, \bm{Z}'$ from the same distribution.

Figures (30)

  • Figure 1: Example of reshuffled empirical loss yielding a worse (left) and better (right) minimizer.
  • Figure 2: Mean true risk (lower is better) of the configuration minimizing the observed objective systematically varied with respect to curvature $m$, correlation strength $\kappa$ of the noise (a larger $\kappa$ implying weaker correlation), and extent of reshuffling $\tau$ (lower $\tau$ increasing reshuffling). A $\tau$ of 1 indicates no reshuffling. Error bars represent standard errors.
  • Figure 3: Average test performance (negative ROC AUC) of the incumbent for XGBoost on dataset albert for increasing $n$ (train-validation sizes, columns). Shaded areas represent standard errors.
  • Figure 4: Average improvement (compared to standard 5-fold CV) with respect to test performance (ROC AUC) of the incumbent over different tasks, learning algorithms and replications separately for increasing $n$ (train-validation sizes, columns). Shaded areas represent standard errors.
  • Figure 5: Average improvement (compared to random search on standard holdout) with respect to test performance (ROC AUC) of the incumbent over tasks, learning algorithms and replications for different $n$ (train-validation sizes, columns). Shaded areas represent standard errors.
  • ...and 25 more figures

Theorems & Definitions (7)

  • Theorem 2.1
  • Theorem 2.2
  • Theorem C.1
  • proof
  • Lemma D.1
  • proof
  • Remark E.1