Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BTTackler: A Diagnosis-based Framework for Efficient Deep Learning Hyperparameter Optimization

Zhongyi Pei, Zhiyao Cen, Yipeng Huang, Chen Wang, Lin Liu, Philip Yu, Mingsheng Long

TL;DR

Bad Trial Tackler (BTTackler), a novel HPO framework that introduces training diagnosis to identify training problems automatically and hence tackles bad trials, is proposed and an open-source Python library that allows users to easily apply BTTackler to automated HPO processes with minimal code changes is released.

Abstract

Hyperparameter optimization (HPO) is known to be costly in deep learning, especially when leveraging automated approaches. Most of the existing automated HPO methods are accuracy-based, i.e., accuracy metrics are used to guide the trials of different hyperparameter configurations amongst a specific search space. However, many trials may encounter severe training problems, such as vanishing gradients and insufficient convergence, which can hardly be reflected by accuracy metrics in the early stages of the training and often result in poor performance. This leads to an inefficient optimization trajectory because the bad trials occupy considerable computation resources and reduce the probability of finding excellent hyperparameter configurations within a time limitation. In this paper, we propose \textbf{Bad Trial Tackler (BTTackler)}, a novel HPO framework that introduces training diagnosis to identify training problems automatically and hence tackles bad trials. BTTackler diagnoses each trial by calculating a set of carefully designed quantified indicators and triggers early termination if any training problems are detected. Evaluations are performed on representative HPO tasks consisting of three classical deep neural networks (DNN) and four widely used HPO methods. To better quantify the effectiveness of an automated HPO method, we propose two new measurements based on accuracy and time consumption. Results show the advantage of BTTackler on two-fold: (1) it reduces 40.33\% of time consumption to achieve the same accuracy comparable to baseline methods on average and (2) it conducts 44.5\% more top-10 trials than baseline methods on average within a given time budget. We also released an open-source Python library that allows users to easily apply BTTackler to automated HPO processes with minimal code changes.

BTTackler: A Diagnosis-based Framework for Efficient Deep Learning Hyperparameter Optimization

TL;DR

Bad Trial Tackler (BTTackler), a novel HPO framework that introduces training diagnosis to identify training problems automatically and hence tackles bad trials, is proposed and an open-source Python library that allows users to easily apply BTTackler to automated HPO processes with minimal code changes is released.

Abstract

Hyperparameter optimization (HPO) is known to be costly in deep learning, especially when leveraging automated approaches. Most of the existing automated HPO methods are accuracy-based, i.e., accuracy metrics are used to guide the trials of different hyperparameter configurations amongst a specific search space. However, many trials may encounter severe training problems, such as vanishing gradients and insufficient convergence, which can hardly be reflected by accuracy metrics in the early stages of the training and often result in poor performance. This leads to an inefficient optimization trajectory because the bad trials occupy considerable computation resources and reduce the probability of finding excellent hyperparameter configurations within a time limitation. In this paper, we propose \textbf{Bad Trial Tackler (BTTackler)}, a novel HPO framework that introduces training diagnosis to identify training problems automatically and hence tackles bad trials. BTTackler diagnoses each trial by calculating a set of carefully designed quantified indicators and triggers early termination if any training problems are detected. Evaluations are performed on representative HPO tasks consisting of three classical deep neural networks (DNN) and four widely used HPO methods. To better quantify the effectiveness of an automated HPO method, we propose two new measurements based on accuracy and time consumption. Results show the advantage of BTTackler on two-fold: (1) it reduces 40.33\% of time consumption to achieve the same accuracy comparable to baseline methods on average and (2) it conducts 44.5\% more top-10 trials than baseline methods on average within a given time budget. We also released an open-source Python library that allows users to easily apply BTTackler to automated HPO processes with minimal code changes.
Paper Structure (18 sections, 5 figures, 6 tables)

This paper contains 18 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. HPO methods
  4. Training Diagnosis
  5. Diagnosis-based HPO Framework
  6. The BTTackler Pipeline
  7. Quality Indicators
  8. What concretely are quality indicators?
  9. How to use quality indicators?
  10. Implementation
  11. Evaluation Method
  12. Experiment
  13. Setup
  14. Effectiveness (RQ1)
  15. Efficiency (RQ2)
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Comparing BTTackler and random search on HPO of a simple CNN on cifar10
  • Figure 2: The framework of BTTackler.
  • Figure 3: The Design of Quality Indicators.
  • Figure 4: The effectiveness of BTTackler-enhanced HPO methods over time.
  • Figure 5: The performance distribution for both bad trials and good trials in the HPO task of Cifar10CNN.

Theorems & Definitions (4)

  • definition 1: Quality Indicator
  • definition 2: Diagnosis-based HPO
  • definition 3: Top10 Hit Ratio
  • definition 4: Time-Saving for Baseline Accuracy