Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Hyperparameter Search into Model-Free AutoML with Context-Free Grammars

Hernán Ceferino Vázquez, Jorge Sanchez, Rafael Carrascosa

TL;DR

This paper extends a grammar-based AutoML framework (GramML) by incorporating hyperparameter search into the production grammar and adapting Monte Carlo Tree Search with pruning and non-parametric selectors to manage the enlarged search space. It introduces and evaluates three selection policies—UTC, BTS, and TPE—in the AutoML grammar context, demonstrating that BTS, in particular, yields superior performance on OpenML-CC18. Experiments reveal that expanding the search space to include hyperparameters significantly improves results over state-of-the-art methods such as AutoSklearn and MOSAIC, with GramML++ variants achieving favorable rankings over time. The work highlights a practical path to more expressive AutoML pipelines and points to future directions like meta-learning integration, resource-aware optimization, and scalable parallelization.

Abstract

Automated Machine Learning (AutoML) has become increasingly popular in recent years due to its ability to reduce the amount of time and expertise required to design and develop machine learning systems. This is very important for the practice of machine learning, as it allows building strong baselines quickly, improving the efficiency of the data scientists, and reducing the time to production. However, despite the advantages of AutoML, it faces several challenges, such as defining the solutions space and exploring it efficiently. Recently, some approaches have been shown to be able to do it using tree-based search algorithms and context-free grammars. In particular, GramML presents a model-free reinforcement learning approach that leverages pipeline configuration grammars and operates using Monte Carlo tree search. However, one of the limitations of GramML is that it uses default hyperparameters, limiting the search problem to finding optimal pipeline structures for the available data preprocessors and models. In this work, we propose an extension to GramML that supports larger search spaces including hyperparameter search. We evaluated the approach using an OpenML benchmark and found significant improvements compared to other state-of-the-art techniques.

Integrating Hyperparameter Search into Model-Free AutoML with Context-Free Grammars

TL;DR

This paper extends a grammar-based AutoML framework (GramML) by incorporating hyperparameter search into the production grammar and adapting Monte Carlo Tree Search with pruning and non-parametric selectors to manage the enlarged search space. It introduces and evaluates three selection policies—UTC, BTS, and TPE—in the AutoML grammar context, demonstrating that BTS, in particular, yields superior performance on OpenML-CC18. Experiments reveal that expanding the search space to include hyperparameters significantly improves results over state-of-the-art methods such as AutoSklearn and MOSAIC, with GramML++ variants achieving favorable rankings over time. The work highlights a practical path to more expressive AutoML pipelines and points to future directions like meta-learning integration, resource-aware optimization, and scalable parallelization.

Abstract

Automated Machine Learning (AutoML) has become increasingly popular in recent years due to its ability to reduce the amount of time and expertise required to design and develop machine learning systems. This is very important for the practice of machine learning, as it allows building strong baselines quickly, improving the efficiency of the data scientists, and reducing the time to production. However, despite the advantages of AutoML, it faces several challenges, such as defining the solutions space and exploring it efficiently. Recently, some approaches have been shown to be able to do it using tree-based search algorithms and context-free grammars. In particular, GramML presents a model-free reinforcement learning approach that leverages pipeline configuration grammars and operates using Monte Carlo tree search. However, one of the limitations of GramML is that it uses default hyperparameters, limiting the search problem to finding optimal pipeline structures for the available data preprocessors and models. In this work, we propose an extension to GramML that supports larger search spaces including hyperparameter search. We evaluated the approach using an OpenML benchmark and found significant improvements compared to other state-of-the-art techniques.
Paper Structure (12 sections, 1 equation, 2 figures, 3 tables, 5 algorithms)

This paper contains 12 sections, 1 equation, 2 figures, 3 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Hyperparameter Search in Grammar-Based AutoML
  5. Upper Confidence Bound applied to Trees (UTC).
  6. Bootstrap Thompson Sampling (BTS).
  7. Tree Parzen Estimator (TPE).
  8. Experiments
  9. Experimental Setup
  10. Ablation Study
  11. Comparison with other techniques
  12. Conclusions and Future Work

Figures (2)

  • Figure 1: Results of the approaches on OpenML-CC18 benchmark.
  • Figure 2: CD plots with Nimenyi post-hoc test