Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How

Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter, Josif Grabocka

TL;DR

This paper proposes a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it, and meta-learn a multi-fidelity performance predictor on the learning curves of this meta-dataset and uses it for fast hyperparameter optimization on new datasets.

Abstract

With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset. In this paper, we propose a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it. Our method transfers knowledge about the performance of many pretrained models with multiple hyperparameter configurations on a series of datasets. To this aim, we evaluated over 20k hyperparameter configurations for finetuning 24 pretrained image classification models on 87 datasets to generate a large-scale meta-dataset. We meta-learn a multi-fidelity performance predictor on the learning curves of this meta-dataset and use it for fast hyperparameter optimization on new datasets. We empirically demonstrate that our resulting approach can quickly select an accurate pretrained model for a new dataset together with its optimal hyperparameters.

Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How

TL;DR

This paper proposes a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it, and meta-learn a multi-fidelity performance predictor on the learning curves of this meta-dataset and uses it for fast hyperparameter optimization on new datasets.

Abstract

With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset. In this paper, we propose a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it. Our method transfers knowledge about the performance of many pretrained models with multiple hyperparameter configurations on a series of datasets. To this aim, we evaluated over 20k hyperparameter configurations for finetuning 24 pretrained image classification models on 87 datasets to generate a large-scale meta-dataset. We meta-learn a multi-fidelity performance predictor on the learning curves of this meta-dataset and use it for fast hyperparameter optimization on new datasets. We empirically demonstrate that our resulting approach can quickly select an accurate pretrained model for a new dataset together with its optimal hyperparameters.
Paper Structure (34 sections, 7 equations, 8 figures, 11 tables, 2 algorithms)

This paper contains 34 sections, 7 equations, 8 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Finetuning Strategies
  4. Model Hubs
  5. HPO, Transfer HPO, and Zero-Shot HPO
  6. Motivation
  7. Quick-Tune: Cost-Efficient Finetuning
  8. Quick-Tune
  9. Performance and Cost Estimators
  10. Cost-sensitive Acquisition Function
  11. Meta-learning the Performance and Cost Estimators
  12. Quick-Tune Meta-Dataset
  13. Quick-Tune Search Space
  14. Meta-Dataset Generation
  15. Experiments and Results
  16. ...and 19 more sections

Figures (8)

  • Figure 1: Ranks of model performances across datasets.
  • Figure 2: The subset of Pareto optimal pretrained models with respect to the predictive accuracy and model size.
  • Figure 3: Comparison against state-of-the-art HPO methods.
  • Figure 4: Comparing Quick-Tune with (+) and without (-) (M)eta-learning and (C)ost-Awareness, and (G)ray-box optimization. We also compare against DyHPO (=QT:-M,-C,+G) and a GP.
  • Figure 5: Varying the model hub size.
  • ...and 3 more figures