Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Black-Box Tuning to Guided Optimization via Hyperparameters Interaction Analysis

Moncef Garouani, Ayah Barhrhouj

TL;DR

Hyperparameter optimization is costly and often lacks guidance on which hyperparameters matter or how they interact across datasets. MetaSHAP combines meta-learning, SHAP-based attribution, and a massive knowledge base to deliver dataset-aware importance scores, interaction insights, and actionable tuning ranges before starting optimization. Empirical evaluation shows that MetaSHAP-guided Bayesian optimization converges faster and with interpretable guidance while reducing the search space, illustrating substantial gains in efficiency and transparency for AutoML workflows. The framework offers a scalable, transferable approach applicable to a range of parameterized learning systems beyond the tested settings.

Abstract

Hyperparameters tuning is a fundamental, yet computationally expensive, step in optimizing machine learning models. Beyond optimization, understanding the relative importance and interaction of hyperparameters is critical to efficient model development. In this paper, we introduce MetaSHAP, a scalable semi-automated eXplainable AI (XAI) method, that uses meta-learning and Shapley values analysis to provide actionable and dataset-aware tuning insights. MetaSHAP operates over a vast benchmark of over 09 millions evaluated machine learning pipelines, allowing it to produce interpretable importance scores and actionable tuning insights that reveal how much each hyperparameter matters, how it interacts with others and in which value ranges its influence is concentrated. For a given algorithm and dataset, MetaSHAP learns a surrogate performance model from historical configurations, computes hyperparameters interactions using SHAP-based analysis, and derives interpretable tuning ranges from the most influential hyperparameters. This allows practitioners not only to prioritize which hyperparameters to tune, but also to understand their directionality and interactions. We empirically validate MetaSHAP on a diverse benchmark of 164 classification datasets and 14 classifiers, demonstrating that it produces reliable importance rankings and competitive performance when used to guide Bayesian optimization.

From Black-Box Tuning to Guided Optimization via Hyperparameters Interaction Analysis

TL;DR

Hyperparameter optimization is costly and often lacks guidance on which hyperparameters matter or how they interact across datasets. MetaSHAP combines meta-learning, SHAP-based attribution, and a massive knowledge base to deliver dataset-aware importance scores, interaction insights, and actionable tuning ranges before starting optimization. Empirical evaluation shows that MetaSHAP-guided Bayesian optimization converges faster and with interpretable guidance while reducing the search space, illustrating substantial gains in efficiency and transparency for AutoML workflows. The framework offers a scalable, transferable approach applicable to a range of parameterized learning systems beyond the tested settings.

Abstract

Hyperparameters tuning is a fundamental, yet computationally expensive, step in optimizing machine learning models. Beyond optimization, understanding the relative importance and interaction of hyperparameters is critical to efficient model development. In this paper, we introduce MetaSHAP, a scalable semi-automated eXplainable AI (XAI) method, that uses meta-learning and Shapley values analysis to provide actionable and dataset-aware tuning insights. MetaSHAP operates over a vast benchmark of over 09 millions evaluated machine learning pipelines, allowing it to produce interpretable importance scores and actionable tuning insights that reveal how much each hyperparameter matters, how it interacts with others and in which value ranges its influence is concentrated. For a given algorithm and dataset, MetaSHAP learns a surrogate performance model from historical configurations, computes hyperparameters interactions using SHAP-based analysis, and derives interpretable tuning ranges from the most influential hyperparameters. This allows practitioners not only to prioritize which hyperparameters to tune, but also to understand their directionality and interactions. We empirically validate MetaSHAP on a diverse benchmark of 164 classification datasets and 14 classifiers, demonstrating that it produces reliable importance rankings and competitive performance when used to guide Bayesian optimization.
Paper Structure (20 sections, 3 equations, 2 figures, 1 table, 1 algorithm)

This paper contains 20 sections, 3 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Theoretical Background
  3. Hyperparameters Optimization (HPO)
  4. Hyperparameters Importance and Interaction
  5. Problem Definition
  6. Related Work
  7. Hyperparameters Importance Estimation
  8. Meta-Learning for HPO Transfer
  9. Methodology
  10. Overview
  11. Step 1: Meta-feature Extraction
  12. Step 2: Similar Tasks Retrieval
  13. Step 3: Surrogate Model Construction
  14. Step 4: Shapley-based HPs Attribution
  15. Extracting Actionable Tuning Insights
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Heatmap displaying SHAP values for each hyperparameter, showing their impact on model output (SVM) on the "ring" dataset. Red indicates a positive contribution, while blue denotes a negative one. The top curve shows the corresponding accuracy evolution. Highlighted vertical bands mark the identified relevant tuning ranges where hyperparameters exhibit the most significant influence on model behavior.
  • Figure 2: Optimization trajectories of standard Bayesian Optimization against the MetaSHAP-Guided BO.