Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastBO: Fast HPO and NAS with Adaptive Fidelity Identification

Jiantong Jiang, Ajmal Mian

TL;DR

A multi-fidelity BO method named FastBO is proposed, which adaptively decides the fidelity for each configuration and efficiently offers strong performance, and the adaptive fidelity identification strategy provides a way to extend any single-fidelity method to the multi-fidelity setting, highlighting its generality and applicability.

Abstract

Hyperparameter optimization (HPO) and neural architecture search (NAS) are powerful in attaining state-of-the-art machine learning models, with Bayesian optimization (BO) standing out as a mainstream method. Extending BO into the multi-fidelity setting has been an emerging research topic, but faces the challenge of determining an appropriate fidelity for each hyperparameter configuration to fit the surrogate model. To tackle the challenge, we propose a multi-fidelity BO method named FastBO, which adaptively decides the fidelity for each configuration and efficiently offers strong performance. The advantages are achieved based on the novel concepts of efficient point and saturation point for each configuration.We also show that our adaptive fidelity identification strategy provides a way to extend any single-fidelity method to the multi-fidelity setting, highlighting its generality and applicability.

FastBO: Fast HPO and NAS with Adaptive Fidelity Identification

TL;DR

A multi-fidelity BO method named FastBO is proposed, which adaptively decides the fidelity for each configuration and efficiently offers strong performance, and the adaptive fidelity identification strategy provides a way to extend any single-fidelity method to the multi-fidelity setting, highlighting its generality and applicability.

Abstract

Hyperparameter optimization (HPO) and neural architecture search (NAS) are powerful in attaining state-of-the-art machine learning models, with Bayesian optimization (BO) standing out as a mainstream method. Extending BO into the multi-fidelity setting has been an emerging research topic, but faces the challenge of determining an appropriate fidelity for each hyperparameter configuration to fit the surrogate model. To tackle the challenge, we propose a multi-fidelity BO method named FastBO, which adaptively decides the fidelity for each configuration and efficiently offers strong performance. The advantages are achieved based on the novel concepts of efficient point and saturation point for each configuration.We also show that our adaptive fidelity identification strategy provides a way to extend any single-fidelity method to the multi-fidelity setting, highlighting its generality and applicability.
Paper Structure (6 sections, 3 figures)

This paper contains 6 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Key Idea of FastBO
  3. Efficient Point and Saturation Point
  4. FastBO Process and Generalization
  5. Experimental Evaluation
  6. Conclusion and Discussion

Figures (3)

  • Figure 1: Main process of FastBO. FastBO involves estimating efficient and saturation points, modeling learning curves, and auxiliary stages of warm-up and post-processing.
  • Figure 2: Anytime performance on the LCBench benchmark.
  • Figure 3: Anytime performance on (a) NAS-Bench-201 and (b) FCNet.

Theorems & Definitions (4)

  • definition thmcounterdefinition: Efficient point
  • Remark 1.
  • definition thmcounterdefinition: Saturation point
  • Remark 2.