Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, Frank Hutter

TL;DR

<3-5 sentence high-level summary> Auto-sklearn 2.0 advances hands-free AutoML by marrying a portfolio-based warmstart with budget-aware evaluation and a learned policy selector that automates high-level design decisions per dataset. It introduces PoSH Auto-sklearn (portfolio + successive halving) for fast, robust performance under tight budgets and then extends to a fully automated AutoML system that selects optimization policies using meta-learning. Empirical results on 39 AutoML benchmarks show substantial reductions in error compared to Auto-sklearn 1.0 and competitive standing against other frameworks, highlighting strong gains in both short and longer time horizons. The work lays a practical foundation for scalable, automatic AutoML and points to future directions in adaptive budgeting, richer meta-features, and broader policy spaces.

Abstract

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour.

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

TL;DR

<3-5 sentence high-level summary> Auto-sklearn 2.0 advances hands-free AutoML by marrying a portfolio-based warmstart with budget-aware evaluation and a learned policy selector that automates high-level design decisions per dataset. It introduces PoSH Auto-sklearn (portfolio + successive halving) for fast, robust performance under tight budgets and then extends to a fully automated AutoML system that selects optimization policies using meta-learning. Empirical results on 39 AutoML benchmarks show substantial reductions in error compared to Auto-sklearn 1.0 and competitive standing against other frameworks, highlighting strong gains in both short and longer time horizons. The work lays a practical foundation for scalable, automatic AutoML and points to future directions in adaptive budgeting, richer meta-features, and broader policy spaces.

Abstract

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour.

Paper Structure

This paper contains 61 sections, 5 theorems, 10 equations, 7 figures, 20 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Time-bounded AutoML
  4. Generalization of AutoML
  5. Part I: Portfolio Successive Halving in PoSH Auto-sklearn
  6. Portfolio Building
  7. Approach
  8. Theoretical Properties of the Greedy Algorithm
  9. Budget Allocation using Successive Halving
  10. Approach
  11. Practical Considerations and Challenge Results
  12. Experimental Setup
  13. Datasets
  14. Meta-data Generation
  15. Other Experimental Details
  16. ...and 46 more sections

Key Result

Proposition 1

Minimizing the test loss of a portfolio $\mathcal{P}$ on a set of datasets $\mathcal{D}_1, \dots, \mathcal{D}_{\vert\mathbf{D}_{\text{meta}}\vert}$, when choosing an ML pipeline from $\mathcal{P}$ for $\mathcal{D}_d$ using holdout or cross-validation based on its performance on $\mathcal{D}_{d,\text

Figures (7)

  • Figure 1: Schematic overview of Auto-sklearn 1.0, PoSH Auto-sklearn, and Auto-sklearn 2.0. Orange rectangular boxes refer to input and output data, while rounded purple boxes denote parts of the AutoML system (surrounded by a green dashed line). The pink, rounded box refers to a human in the loop required for manual design decisions. The newer AutoML systems simplify the usage of Auto-sklearn and reduce the required user input. We describe PoSH Auto-sklearn in Section \ref{['sec:partI']} and give a schematic overview in Figure \ref{['fig:schema_posh']}. Similarly, we describe Auto-sklearn 2.0 in Section \ref{['sec:partII']} and provide a schematic overview in Figure \ref{['fig:schema_main']}.
  • Figure 2: Schematic Overview of PoSH Auto-sklearn with the offline portfolio building phase (TR1-TR3) above and the test phase (TE1-TE2) below the dashed line. Rounded, purple boxes refer to computational steps while rectangular, orange boxes depict the input data to the AutoML system.
  • Figure 3: Distribution of meta and test datasets. We visualize each dataset w.r.t. its meta-features and highlight the datasets outside our meta distribution using black crosses.
  • Figure 4: Final balanced error rate of BO using different model selection strategies averaged across $10$ repetitions. Top row: Results for a optimization budget of $10$ minutes on two different datasets. Bottom row: Results for a optimization budget of $10$ and $60$ minutes on the same dataset.
  • Figure 5: Schematic overview of the proposed Auto-sklearn 2.0 system with the training phase (TR1--TR6) above and the test phase (MtL1--MtL2&TE1--TE2) below the dashed line. Rounded, purple boxes refer to computational steps, while rectangular, orange boxes depict the input data to the AutoML system.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Proposition 1
  • Corollary 1
  • Corollary 2
  • Corollary 3
  • Definition 1
  • Definition 2
  • Definition 3
  • Proposition 2