Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyperparameter Optimization via Interacting with Probabilistic Circuits

Jonas Seng, Fabrizio Ventola, Zhongjie Yu, Kristian Kersting

TL;DR

This work tackles interactive hyperparameter optimization by replacing acquisition-based guidance with a tractable probabilistic-circuit surrogate that supports exact conditioning. IBO-HPC uses conditional sampling to generate candidates that honor user beliefs, optionally weighting them with a user prior, and avoids inner-loop optimization of acquisition functions. The authors prove that the interactive policy is feedback-adhering and globally convergent, and they provide an EI-based convergence analysis under Gaussian-leaf PCs. Empirically, IBO-HPC is competitive with strong HPO baselines and often outperforms interactive baselines when user knowledge is provided, achieving substantial speed-ups and demonstrating robustness to misleading feedback. The approach promises practical gains for human-in-the-loop AutoML, while outlining limitations and directions for future work such as transfer learning across tasks.

Abstract

Despite the growing interest in designing truly interactive hyperparameter optimization (HPO) methods, to date, only a few allow to include human feedback. Existing interactive Bayesian optimization (BO) methods incorporate human beliefs by weighting the acquisition function with a user-defined prior distribution. However, in light of the non-trivial inner optimization of the acquisition function prevalent in BO, such weighting schemes do not always accurately reflect given user beliefs. We introduce a novel BO approach leveraging tractable probabilistic models named probabilistic circuits (PCs) as a surrogate model. PCs encode a tractable joint distribution over the hybrid hyperparameter space and evaluation scores. They enable exact conditional inference and sampling. Based on conditional sampling, we construct a novel selection policy that enables an acquisition function-free generation of candidate points (thereby eliminating the need for an additional inner-loop optimization) and ensures that user beliefs are reflected accurately in the selection policy. We provide a theoretical analysis and an extensive empirical evaluation, demonstrating that our method achieves state-of-the-art performance in standard HPO and outperforms interactive BO baselines in interactive HPO.

Hyperparameter Optimization via Interacting with Probabilistic Circuits

TL;DR

This work tackles interactive hyperparameter optimization by replacing acquisition-based guidance with a tractable probabilistic-circuit surrogate that supports exact conditioning. IBO-HPC uses conditional sampling to generate candidates that honor user beliefs, optionally weighting them with a user prior, and avoids inner-loop optimization of acquisition functions. The authors prove that the interactive policy is feedback-adhering and globally convergent, and they provide an EI-based convergence analysis under Gaussian-leaf PCs. Empirically, IBO-HPC is competitive with strong HPO baselines and often outperforms interactive baselines when user knowledge is provided, achieving substantial speed-ups and demonstrating robustness to misleading feedback. The approach promises practical gains for human-in-the-loop AutoML, while outlining limitations and directions for future work such as transfer learning across tasks.

Abstract

Despite the growing interest in designing truly interactive hyperparameter optimization (HPO) methods, to date, only a few allow to include human feedback. Existing interactive Bayesian optimization (BO) methods incorporate human beliefs by weighting the acquisition function with a user-defined prior distribution. However, in light of the non-trivial inner optimization of the acquisition function prevalent in BO, such weighting schemes do not always accurately reflect given user beliefs. We introduce a novel BO approach leveraging tractable probabilistic models named probabilistic circuits (PCs) as a surrogate model. PCs encode a tractable joint distribution over the hybrid hyperparameter space and evaluation scores. They enable exact conditional inference and sampling. Based on conditional sampling, we construct a novel selection policy that enables an acquisition function-free generation of candidate points (thereby eliminating the need for an additional inner-loop optimization) and ensures that user beliefs are reflected accurately in the selection policy. We provide a theoretical analysis and an extensive empirical evaluation, demonstrating that our method achieves state-of-the-art performance in standard HPO and outperforms interactive BO baselines in interactive HPO.

Paper Structure

This paper contains 41 sections, 5 theorems, 13 equations, 17 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Interactive Bayesian Optimization
  4. Probabilistic Circuits (PCs)
  5. Interactive Hyperparameter Optimization
  6. Interactivity in Bayesian HPO
  7. Interactive Bayesian Optimization with Hyperparameter Probabilistic Circuits
  8. Method
  9. Theoretical Analysis
  10. Experimental Evaluation
  11. Experimental Setup
  12. User Interactions
  13. (Q1) IBO-HPC is Competitive in HPO & NAS
  14. (Q2, Q3) IBO-HPC is Interactive and Resilient
  15. Beneficial Interactions
  16. ...and 26 more sections

Key Result

Proposition 1

Assume a search space $\bm{\Theta}$ over hyperparameters $\bm{\mathcal{H}}$, a PC $s$, user knowledge $\mathcal{K} \in \bm{\mathcal{K}}$ in form of a prior $q$ over $\hat{\bm{\mathcal{H}}} \subset \bm{\mathcal{H}}$ s.t. the marginal over $\hat{\bm{\mathcal{H}}}$ of $s$ conditioned on $f^*$ is differ

Figures (17)

  • Figure 1: Interactive Bayesian Hyperparameter Optimization. (Left) We devise an interactive BO method by employing PCs as surrogates encoding a joint distribution over hyperparameters and evaluation scores (omitted for clarity). PCs allow users to directly condition the surrogate on their beliefs during tractable candidate generation, thereby reflecting user knowledge accurately. (Right) Accurately reflecting user beliefs is crucial for interactive HPO to fully leverage user knowledge. In contrast to $\pi$BO and BOPrO, IBO-HPC (our method) precisely reflects the user prior provided for hyperparameter $R$ (resolution). See App. \ref{['app:motivation']} for details.
  • Figure 2: IBO-HPC outperforms state of the art. For 5/5 tasks across three challenging benchmarks, IBO-HPC is competitive with strong baselines when no user knowledge is provided. When beneficial user beliefs are provided ( ), after 5 ( ) or after 10 iterations ( ), it outperforms all competitors w.r.t. convergence and/or solution quality on 4/5 tasks.
  • Figure 3: IBO-HPC recovers from misleading interactions. IBO-HPC automatically recovers from ( ) misleading feedback provided as point values at the 5th iteration of the search (1st ). Also, when providing harmful and beneficial beliefs alternatively ( / ), IBO-HPC ( ) catches up with or outperforms $\pi$BO ( ) and BOPrO ( ) in 4/5 cases.
  • Figure 4: IBO-HPC achieves considerable runtime improvement with beneficial interactions (2-10$\times$ faster).
  • Figure 5: IBO-HPC reflects user priors as specified. In contrast to other weighting scheme based methods like $\pi$BO and BOPrO, IBO-HPC reflects the user prior as specified in its selection policy.
  • ...and 12 more figures

Theorems & Definitions (12)

  • Definition 1: Hyperparameter optimization (HPO)
  • Definition 2: Feedback-Adhering Interactive Policy
  • Remark 1
  • Proposition 1: IBO-HPC Policy is feedback-adhering interactive
  • Proposition 2: IBO-HPC is a global optimizer
  • Proposition 3: Convergence of IBO-HPC
  • Definition 3
  • proof
  • Proposition 4: IBO-HPC minimizes Simple Regret
  • proof
  • ...and 2 more