Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

Kaito Baba, Yoshihiko Ozaki, Shuhei Watanabe

TL;DR

This paper tackles hyperparameter importance (HPI) estimation in conditional search spaces where hyperparameters may be inactive or change domains depending on others. It introduces condPED-ANOVA, a conditional extension of PED-ANOVA that defines conditional local HPI via within-regime variance and derives a closed-form estimator using the Pearson divergence between regime-specific one-dimensional PDFs. The approach yields meaningful, interpretable importances that reflect the underlying conditional structure, avoiding misleading attributions caused by naive extensions. Empirically, condPED-ANOVA is fast and robust across conditional activation and regime-switching scenarios, supporting reliable HPI analysis in AutoML and HPO pipelines.

Abstract

We propose conditional PED-ANOVA (condPED-ANOVA), a principled framework for estimating hyperparameter importance (HPI) in conditional search spaces, where the presence or domain of a hyperparameter can depend on other hyperparameters. Although the original PED-ANOVA provides a fast and efficient way to estimate HPI within the top-performing regions of the search space, it assumes a fixed, unconditional search space and therefore cannot properly handle conditional hyperparameters. To address this, we introduce a conditional HPI for top-performing regions and derive a closed-form estimator that accurately reflects conditional activation and domain changes. Experiments show that naive adaptations of existing HPI estimators yield misleading or uninterpretable importance estimates in conditional settings, whereas condPED-ANOVA consistently provides meaningful importances that reflect the underlying conditional structure.

Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

TL;DR

This paper tackles hyperparameter importance (HPI) estimation in conditional search spaces where hyperparameters may be inactive or change domains depending on others. It introduces condPED-ANOVA, a conditional extension of PED-ANOVA that defines conditional local HPI via within-regime variance and derives a closed-form estimator using the Pearson divergence between regime-specific one-dimensional PDFs. The approach yields meaningful, interpretable importances that reflect the underlying conditional structure, avoiding misleading attributions caused by naive extensions. Empirically, condPED-ANOVA is fast and robust across conditional activation and regime-switching scenarios, supporting reliable HPI analysis in AutoML and HPO pipelines.

Abstract

We propose conditional PED-ANOVA (condPED-ANOVA), a principled framework for estimating hyperparameter importance (HPI) in conditional search spaces, where the presence or domain of a hyperparameter can depend on other hyperparameters. Although the original PED-ANOVA provides a fast and efficient way to estimate HPI within the top-performing regions of the search space, it assumes a fixed, unconditional search space and therefore cannot properly handle conditional hyperparameters. To address this, we introduce a conditional HPI for top-performing regions and derive a closed-form estimator that accurately reflects conditional activation and domain changes. Experiments show that naive adaptations of existing HPI estimators yield misleading or uninterpretable importance estimates in conditional settings, whereas condPED-ANOVA consistently provides meaningful importances that reflect the underlying conditional structure.
Paper Structure (53 sections, 4 theorems, 26 equations, 19 figures, 3 algorithms)

This paper contains 53 sections, 4 theorems, 26 equations, 19 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Variance-Based HPI and f-ANOVA
  4. PED-ANOVA
  5. Conditional Hyperparameters in HPO
  6. HPI under Conditional Search Spaces
  7. Preliminaries
  8. Notations
  9. Local HPI
  10. PED-ANOVA
  11. condPED-ANOVA: Hyperparameter Importance in Conditional Search Spaces
  12. Regime-Based Representation of Conditional Hyperparameters
  13. Conditional Local HPI via Within-Regime Variance
  14. condPED-ANOVA
  15. Numerical Experiments
  16. ...and 38 more sections

Key Result

theorem 1

Let $0<\gamma^\prime<\gamma\le 1$. The within-regime local marginal variance for the $d$-th hyperparameter at level $\gamma$, computed using the indicator function $b_{\gamma^\prime} \coloneqq \mathbf{1}\{x\in\mathcal{X}_{\gamma^\prime}\}$, is given by: By normalizing the variance across all hyperparameters as defined in eq:cond-local-hpi-def, we obtain the conditional local HPI for the $d$-th hy

Figures (19)

  • Figure 1: Conditional activation (\ref{['eq:toy-objective-conditional']}) with disjoint domains (\ref{['eq:toy-domains-disjoint']})
  • Figure 2: Conditional activation (\ref{['eq:toy-objective-conditional']}) with overlapping domains (\ref{['eq:toy-domains-overlap']})
  • Figure 3: Regime-dependent domains (\ref{['eq:toy-objective-regime-dependent']} and \ref{['eq:toy-domain-regime-dependent']})
  • Figure 5: Baseline HPIs computed with naive extensions of existing methods for the synthetic objective with conditional activation (\ref{['eq:toy-objective-conditional']}) under the disjoint domain setting (\ref{['eq:toy-domains-disjoint']}). "Filtering" computes HPI on the subset of samples where the target hyperparameter is active, whereas "Imputation" assigns a default value (the domain midpoint) to inactive samples before computing HPI. The lines and bars denote the mean, and the shaded regions and error bars denote the standard error, both computed over ten independent runs with different random seeds.
  • Figure 6: Conditional activation (\ref{['eq:toy-objective-conditional']}) with disjoint domains (\ref{['eq:toy-domains-disjoint']})
  • ...and 14 more figures

Theorems & Definitions (5)

  • definition 1: Conditional Local HPI
  • theorem 1: condPED-ANOVA
  • theorem 2: condPED-ANOVA (Restated)
  • theorem 3: Leakage of gating effects under the standard local HPI
  • lemma 1: Regime-wise decomposition of Pearson divergence