Reproducibility Study of Large Language Model Bayesian Optimization

TL;DR

This reproducibility study evaluates LLAMBO, a prompting-based Bayesian optimization framework that uses large language models as discriminative surrogates and text-only acquisition strategies, by re-running Bayesmark and HPOBench experiments with the open-weight Llama 3.1 $70$B backbone instead of GPT-3.5. The authors confirm that contextual warmstarting from textual problem and hyperparameter descriptions substantially improves early regret and reduces run-to-run variance, while the LLAMBO surrogate benefits from cross-task semantic priors despite weaker single-task predictive performance relative to GP/SMAC. Ablations removing textual context degrade predictive accuracy and calibration, and LLAMBO’s candidate sampler consistently yields higher-quality, more diverse proposals than TPE or random sampling; smaller backbones (Gemma 27B, Llama 3.1 8B) produce unstable or invalid predictions, indicating a capacity threshold for reliable surrogate behavior. Overall, LLAMBO remains robust to changes in the language-model backbone and demonstrates effective performance with Llama 3.1 $70$B; code and configurations are available at GitHub for full reproducibility.

Abstract

In this reproducibility study, we revisit the LLAMBO framework of Daxberger et al. (2024), a prompting-based Bayesian optimization (BO) method that uses large language models as discriminative surrogates and acquisition optimizers via text-only interactions. We replicate the core Bayesmark and HPOBench experiments under the original evaluation protocol, but replace GPT-3.5 with the open-weight Llama 3.1 70B model used for all text encoding components. Our results broadly confirm the main claims of LLAMBO. Contextual warm starting via textual problem and hyperparameter descriptions substantially improves early regret behaviour and reduces variance across runs. LLAMBO's discriminative surrogate is weaker than GP or SMAC as a pure single task regressor, yet benefits from cross task semantic priors induced by the language model. Ablations that remove textual context markedly degrade predictive accuracy and calibration, while the LLAMBO candidate sampler consistently generates higher quality and more diverse proposals than TPE or random sampling. Experiments with smaller backbones (Gemma 27B, Llama 3.1 8B) yield unstable or invalid predictions, suggesting insufficient capacity for reliable surrogate behaviour. Overall, our study shows that the LLAMBO architecture is robust to changing the language model backbone and remains effective when instantiated with Llama 3.1 70B.

Figures (8)

• Figure 1: Overview of the LLAMBO prompting pipeline. A dataset description (Data Card), model specification (Model Card), prior observations, and task instructions are combined into structured prompts that guide the LLM through (0) zero-shot warmstarting, (1) candidate generation, and (2) surrogate-based performance estimation. Each proposed hyperparameter configuration is evaluated by the objective function, added to the data history, and reintroduced into subsequent prompts, enabling iterative refinement of the search strategy purely through natural-language interactions.
• Figure 2: Baseline Regret on Bayesmark Public Tasks and Private + Synthetic Tasks.
• Figure 3: Warmstarting regret curves. Average simple regret over trials for classical space-filling methods (Random, Sobol, LHCube) and contextual warmstarts (No Context, Partial Context, Full Context).
• Figure 4: Correlation structure of initial designs. Pairwise correlation matrices of normalized hyperparameters for different warmstarting strategies on a representative dataset/model.
• Figure 5: Diversity of initial designs. Generalized variance of normalized hyperparameters for each warmstarting strategy (higher is more diverse).