Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LB-MCTS: Synergizing Large Language Models and Bayesian Optimization for Efficient CASH

Beicheng Xu, Weitong Qian, Lingching Tung, Yupeng Lu, Bin Cui

TL;DR

LB-MCTS addresses CASH by integrating Large Language Models with Bayesian Optimization inside a Monte Carlo Tree Search framework. It introduces Selective Tuning Memory to provide algorithm-specific context, a tree-structured search to balance inter- and intra-algorithm exploration, and a dynamic switching mechanism that blends LLM- and BO-proposed configurations. Empirical results across 104 AMLB datasets show LB-MCTS outperforming BO, LLM, and hybrid baselines while maintaining cost efficiency, illustrating improved sample efficiency and robustness in high-dimensional AutoML spaces. The approach holds practical impact for deploying AutoML systems that combine semantic priors with rigorous optimization, enabling faster convergence and better generalization in diverse tasks.

Abstract

To lower the expertise barrier in machine learning, the AutoML community has focused on the CASH problem, a fundamental challenge that automates the process of algorithm selection and hyperparameter tuning. While traditional methods like Bayesian Optimization (BO) struggle with cold-start issues, Large Language Models (LLMs) can mitigate these via semantic priors. However, existing LLM-based optimizers generalize poorly to the high-dimensional, structured CASH space. We propose LB-MCTS, a framework synergizing LLMs and BO within a Monte Carlo Tree Search structure. It maximizes LLM reasoning with Selective Tuning Memory (STM) and explicit exploration-exploitation trade-off. It combines the strengths of both paradigms by dynamically shifting from LLM-driven to BO-driven proposals as data accumulates. Experiments on 104 AMLB datasets demonstrate the superiority of LB-MCTS over the competitive baselines.

LB-MCTS: Synergizing Large Language Models and Bayesian Optimization for Efficient CASH

TL;DR

LB-MCTS addresses CASH by integrating Large Language Models with Bayesian Optimization inside a Monte Carlo Tree Search framework. It introduces Selective Tuning Memory to provide algorithm-specific context, a tree-structured search to balance inter- and intra-algorithm exploration, and a dynamic switching mechanism that blends LLM- and BO-proposed configurations. Empirical results across 104 AMLB datasets show LB-MCTS outperforming BO, LLM, and hybrid baselines while maintaining cost efficiency, illustrating improved sample efficiency and robustness in high-dimensional AutoML spaces. The approach holds practical impact for deploying AutoML systems that combine semantic priors with rigorous optimization, enabling faster convergence and better generalization in diverse tasks.

Abstract

To lower the expertise barrier in machine learning, the AutoML community has focused on the CASH problem, a fundamental challenge that automates the process of algorithm selection and hyperparameter tuning. While traditional methods like Bayesian Optimization (BO) struggle with cold-start issues, Large Language Models (LLMs) can mitigate these via semantic priors. However, existing LLM-based optimizers generalize poorly to the high-dimensional, structured CASH space. We propose LB-MCTS, a framework synergizing LLMs and BO within a Monte Carlo Tree Search structure. It maximizes LLM reasoning with Selective Tuning Memory (STM) and explicit exploration-exploitation trade-off. It combines the strengths of both paradigms by dynamically shifting from LLM-driven to BO-driven proposals as data accumulates. Experiments on 104 AMLB datasets demonstrate the superiority of LB-MCTS over the competitive baselines.
Paper Structure (40 sections, 1 theorem, 8 equations, 16 figures, 3 tables, 1 algorithm)

This paper contains 40 sections, 1 theorem, 8 equations, 16 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary and Related Work
  3. CASH Problem and Traditional Methods
  4. LLM-based Optimizer
  5. Monte-Carlo Tree Search
  6. Method
  7. Tree Structured Search for CASH
  8. Algorithm Selection
  9. Hyperparameter Optimization
  10. BO proposer
  11. LLM proposer
  12. Dynamic Proposer Selection
  13. Algorithm Summary
  14. Discussion
  15. Experiment
  16. ...and 25 more sections

Key Result

Theorem 3.1

If the probability of selecting the BO proposer is bounded below by a constant $\epsilon > 0$ (i.e., $P_{\text{BO}} \ge \epsilon$), LB-MCTS converges to the global optimum almost surely as the number of iterations $T \to \infty$:

Figures (16)

  • Figure 1: Convergence of LLM-based methods on 104 datasets.
  • Figure 2: Tree-structured search of LB-MCTS. See Appendix \ref{['app:visualization_of_search_process']} for a concrete case of the optimization on a real-world dataset.
  • Figure 3: MCTS search of the LLM proposer in LB-MCTS.
  • Figure 4: Average Performance of 9 methods across 104 datasets.
  • Figure 5: Resource allocation distribution across 104 datasets.
  • ...and 11 more figures

Theorems & Definitions (2)

  • Theorem 3.1: Global Convergence
  • proof