Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Sequential Statistical Tests for Efficient Hyperparameter Tuning

Philip Buczak, Andreas Groll, Markus Pauly, Jakob Rehof, Daniel Horn

TL;DR

The sequential random search (SQRS) is proposed which extends the regular random search algorithm by a sequential testing procedure aimed at detecting and eliminating inferior parameter configurations early.

Abstract

Hyperparameter tuning is one of the the most time-consuming parts in machine learning. Despite the existence of modern optimization algorithms that minimize the number of evaluations needed, evaluations of a single setting may still be expensive. Usually a resampling technique is used, where the machine learning method has to be fitted a fixed number of k times on different training datasets. The respective mean performance of the k fits is then used as performance estimator. Many hyperparameter settings could be discarded after less than k resampling iterations if they are clearly inferior to high-performing settings. However, resampling is often performed until the very end, wasting a lot of computational effort. To this end, we propose the Sequential Random Search (SQRS) which extends the regular random search algorithm by a sequential testing procedure aimed at detecting and eliminating inferior parameter configurations early. We compared our SQRS with regular random search using multiple publicly available regression and classification datasets. Our simulation study showed that the SQRS is able to find similarly well-performing parameter settings while requiring noticeably fewer evaluations. Our results underscore the potential for integrating sequential tests into hyperparameter tuning.

Using Sequential Statistical Tests for Efficient Hyperparameter Tuning

TL;DR

The sequential random search (SQRS) is proposed which extends the regular random search algorithm by a sequential testing procedure aimed at detecting and eliminating inferior parameter configurations early.

Abstract

Hyperparameter tuning is one of the the most time-consuming parts in machine learning. Despite the existence of modern optimization algorithms that minimize the number of evaluations needed, evaluations of a single setting may still be expensive. Usually a resampling technique is used, where the machine learning method has to be fitted a fixed number of k times on different training datasets. The respective mean performance of the k fits is then used as performance estimator. Many hyperparameter settings could be discarded after less than k resampling iterations if they are clearly inferior to high-performing settings. However, resampling is often performed until the very end, wasting a lot of computational effort. To this end, we propose the Sequential Random Search (SQRS) which extends the regular random search algorithm by a sequential testing procedure aimed at detecting and eliminating inferior parameter configurations early. We compared our SQRS with regular random search using multiple publicly available regression and classification datasets. Our simulation study showed that the SQRS is able to find similarly well-performing parameter settings while requiring noticeably fewer evaluations. Our results underscore the potential for integrating sequential tests into hyperparameter tuning.
Paper Structure (7 sections, 6 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 7 sections, 6 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Machine Learning and Hyperparameter Tuning
  3. Datasets and Machine Learning Algorithms
  4. Determining an Appropriate Sequential Test
  5. Integrating the Sequential Test into Random Search
  6. Simulation Study
  7. Discussion

Figures (5)

  • Figure 1: Cramér-von-Mises criterion values for different distribution classes in the regression case.
  • Figure 2: Cramér-von-Mises criterion values for different distribution classes with individual additive shifts $c$ in the classification case.
  • Figure 3: Performance ratios of SQRS/random search in the case of different solutions (i.e., performance ratio SQRS/random search $\neq$ 1) for regression (A-D) and classification (E-H) SQRS settings.
  • Figure 4: Ratios of evaluations needed for SQRS/random search for regression (A-D) and classification (E-H) SQRS settings.
  • Figure 5: Exemplary parallel variant of the SQRS for parameter configurations $P_1, \dots, P_{2^k}$.