Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter Optimization with Conscious Allocation (POCA)

Joshua Inman, Tanmay Khandait, Giulia Pedrielli, Lalitha Sankar

TL;DR

POCA tackles the efficiency challenge of hyperparameter optimization by introducing an adaptive budget-allocation strategy within a Hyperband framework, guided by a Tree Parzen Estimator. It shortens early hyperbands to save resources while allocating more budget to later, more promising configurations as the surrogate improves, and it leverages all past data in a single KDE-based surrogate. Empirical results on a Counting Ones toy function and MNIST show POCA finds strong configurations faster than BOHB and achieves competitive final performance. This approach enhances early exploration and data efficiency, with practical impact for Auto-ML workflows where budget is a critical bottleneck.

Abstract

The performance of modern machine learning algorithms depends upon the selection of a set of hyperparameters. Common examples of hyperparameters are learning rate and the number of layers in a dense neural network. Auto-ML is a branch of optimization that has produced important contributions in this area. Within Auto-ML, hyperband-based approaches, which eliminate poorly-performing configurations after evaluating them at low budgets, are among the most effective. However, the performance of these algorithms strongly depends on how effectively they allocate the computational budget to various hyperparameter configurations. We present the new Parameter Optimization with Conscious Allocation (POCA), a hyperband-based algorithm that adaptively allocates the inputted budget to the hyperparameter configurations it generates following a Bayesian sampling scheme. We compare POCA to its nearest competitor at optimizing the hyperparameters of an artificial toy function and a deep neural network and find that POCA finds strong configurations faster in both settings.

Parameter Optimization with Conscious Allocation (POCA)

TL;DR

POCA tackles the efficiency challenge of hyperparameter optimization by introducing an adaptive budget-allocation strategy within a Hyperband framework, guided by a Tree Parzen Estimator. It shortens early hyperbands to save resources while allocating more budget to later, more promising configurations as the surrogate improves, and it leverages all past data in a single KDE-based surrogate. Empirical results on a Counting Ones toy function and MNIST show POCA finds strong configurations faster than BOHB and achieves competitive final performance. This approach enhances early exploration and data efficiency, with practical impact for Auto-ML workflows where budget is a critical bottleneck.

Abstract

The performance of modern machine learning algorithms depends upon the selection of a set of hyperparameters. Common examples of hyperparameters are learning rate and the number of layers in a dense neural network. Auto-ML is a branch of optimization that has produced important contributions in this area. Within Auto-ML, hyperband-based approaches, which eliminate poorly-performing configurations after evaluating them at low budgets, are among the most effective. However, the performance of these algorithms strongly depends on how effectively they allocate the computational budget to various hyperparameter configurations. We present the new Parameter Optimization with Conscious Allocation (POCA), a hyperband-based algorithm that adaptively allocates the inputted budget to the hyperparameter configurations it generates following a Bayesian sampling scheme. We compare POCA to its nearest competitor at optimizing the hyperparameters of an artificial toy function and a deep neural network and find that POCA finds strong configurations faster in both settings.
Paper Structure (19 sections, 2 equations, 4 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 2 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model-free Algorithms
  4. Bayesian Optimization
  5. Multi-fidelity Algorithms
  6. Hybrid Approaches
  7. Problem Statement
  8. Hyperband
  9. Contributions
  10. PARAMETER OPTIMIZATION with CONSCIOUS ALLOCATION
  11. Hyperband
  12. Budget Allocation
  13. Adaptive Configuration Selection
  14. Surrogate Update
  15. Numerical Analysis
  16. ...and 4 more sections

Figures (4)

  • Figure 1: POCA flowchart.
  • Figure 2: Hyperbands scheduled by BOHB and POCA for the example in Section \ref{['Budget Allocation']}. Each horizontal bar represents a single configuration.
  • Figure 3: Average loss of the best configuration found by POCA and BOHB for the Counting Ones experiment with 95% confidence intervals.
  • Figure 4: Accuracy plots for MNIST.