Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence

Joshua Inman, Tanmay Khandait, Lalitha Sankar, Giulia Pedrielli

TL;DR

POCAII tackles budget-constrained hyperparameter optimization by decoupling configuration search from evaluation, allowing broad early exploration followed by focused, evidence-driven resource allocation. It combines Tree-structured Parzen Estimation sampling with ARIMA-based forecasting of loss trajectories to guide when and which configurations to re-evaluate, adapting budgets through a two-phase process driven by observed performance. The work contributes a novel two-phase POCAII architecture, formal analysis showing asymptotic equivalence to Hyperband, and empirical validation on the YAHPO Gym and MNIST benchmarks, demonstrating strong performance and low variance in low-budget regimes. This approach offers a scalable, robust HPO framework well-suited to real-world problems where expensive models and limited compute are the norm.

Abstract

In this paper we propose for the first time the hyperparameter optimization (HPO) algorithm POCAII. POCAII differs from the Hyperband and Successive Halving literature by explicitly separating the search and evaluation phases and utilizing principled approaches to exploration and exploitation principles during both phases. Such distinction results in a highly flexible scheme for managing a hyperparameter optimization budget by focusing on search (i.e., generating competing configurations) towards the start of the HPO process while increasing the evaluation effort as the HPO comes to an end. POCAII was compared to state of the art approaches SMAC, BOHB and DEHB. Our algorithm shows superior performance in low-budget hyperparameter optimization regimes. Since many practitioners do not have exhaustive resources to assign to HPO, it has wide applications to real-world problems. Moreover, the empirical evidence showed how POCAII demonstrates higher robustness and lower variance in the results. This is again very important when considering realistic scenarios with extremely expensive models to train.

POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence

TL;DR

POCAII tackles budget-constrained hyperparameter optimization by decoupling configuration search from evaluation, allowing broad early exploration followed by focused, evidence-driven resource allocation. It combines Tree-structured Parzen Estimation sampling with ARIMA-based forecasting of loss trajectories to guide when and which configurations to re-evaluate, adapting budgets through a two-phase process driven by observed performance. The work contributes a novel two-phase POCAII architecture, formal analysis showing asymptotic equivalence to Hyperband, and empirical validation on the YAHPO Gym and MNIST benchmarks, demonstrating strong performance and low variance in low-budget regimes. This approach offers a scalable, robust HPO framework well-suited to real-world problems where expensive models and limited compute are the norm.

Abstract

In this paper we propose for the first time the hyperparameter optimization (HPO) algorithm POCAII. POCAII differs from the Hyperband and Successive Halving literature by explicitly separating the search and evaluation phases and utilizing principled approaches to exploration and exploitation principles during both phases. Such distinction results in a highly flexible scheme for managing a hyperparameter optimization budget by focusing on search (i.e., generating competing configurations) towards the start of the HPO process while increasing the evaluation effort as the HPO comes to an end. POCAII was compared to state of the art approaches SMAC, BOHB and DEHB. Our algorithm shows superior performance in low-budget hyperparameter optimization regimes. Since many practitioners do not have exhaustive resources to assign to HPO, it has wide applications to real-world problems. Moreover, the empirical evidence showed how POCAII demonstrates higher robustness and lower variance in the results. This is again very important when considering realistic scenarios with extremely expensive models to train.
Paper Structure (23 sections, 5 theorems, 26 equations, 4 figures, 6 tables, 2 algorithms)

This paper contains 23 sections, 5 theorems, 26 equations, 4 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Model-free approaches
  4. Bayesian Optimization approaches
  5. Multifidelity approaches
  6. Problem Formulation
  7. POCAII Method
  8. Background
  9. Tree Parzen Estimation.
  10. POCAII Search Phase
  11. POCAII Evaluation Phase
  12. CASE (a): $\Gamma_k = \emptyset$.
  13. CASE (b): $|\Gamma_k| > 0$.
  14. Implementation Guidelines
  15. Asymptotic Behavior Analysis
  16. ...and 8 more sections

Key Result

Proposition 5.3

Let $B$ be total budget and $\eta > 1$ Hyperband's elimination rate. For POCAII with sampling distribution: where $R_k = B - \sum_{t=1}^{k-1}\beta_t$, then: matches Hyperband's uniform bracket initialization.

Figures (4)

  • Figure 1: Algorithm flowchart of POCAII. When the algorithm stops it first uses the remainder epochs allocating them proportionally to the response improvement predicted for each configuration. Finally the configuration with the best associated loss value is returned as incumbent.
  • Figure 2: Average ranks of POCAII, SMAC, DEHB, and BOHB on the YAHPO Gym LCNet Scenario
  • Figure 3: Average Test Accuracy over $90$ Replications against Epochs for representative Datasets
  • Figure 4: Boxplot of Test Accuracies of the Best-Found Configurations of the POCAII and SMAC algorithms after various budgets of HPO.

Theorems & Definitions (6)

  • Proposition 5.3: Sampling Correspondence
  • proof
  • Proposition 5.4: Infinite Resampling
  • Lemma 5.5: POCA Satisfies Resampling
  • Lemma 5.6: Budget Schedule Congruence
  • Theorem 5.7: Constructive Equivalence