GPTOpt: Towards Efficient LLM-Based Black-Box Optimization

TL;DR

Efficient global optimization of expensive, derivative-free black-box functions is challenging under tight evaluation budgets. GPTOpt fine-tunes an LLM (Llama 3.2 3B) via LoRA on a large synthetic dataset of optimization trajectories generated from diverse BO parameterizations to acquire optimization capabilities and enable zero-shot generalization. It encodes optimization trajectories into text, applies adaptive objective scaling, and uses an acquisition step based on multiple forward passes to select the next point with improved objective value. The method outperforms traditional Bayesian optimization and related baselines on both in-distribution holdout benchmarks and out-of-distribution suites (BBOB and VLSE) up to 10 dimensions, demonstrating robust generalization and the potential for plug-and-play optimization with pre-trained language models.

Abstract

Global optimization of expensive, derivative-free black-box functions demands extreme sample efficiency. Classical methods such as Bayesian Optimization (BO) can be effective, but they often require careful parameter tuning to each application domain. At the same time, Large Language Models (LLMs) have shown broad capabilities, yet state-of-the-art models remain limited in solving continuous black-box optimization tasks. We introduce GPTOpt, an LLM-based optimization method that equips LLMs with continuous black-box optimization capabilities. By fine-tuning large language models on extensive synthetic datasets derived from diverse BO parameterizations, GPTOpt leverages LLM pre-training to generalize across optimization tasks. On a variety of black-box optimization benchmarks, GPTOpt surpasses traditional optimizers, highlighting the capacity of LLMs for advanced numerical reasoning and introducing a flexible framework for global optimization without parameter tuning.

Figures (7)

• Figure 1: Performance comparison of best BO-based method to individual BO methods and SOTA LLM-based optimization over 10 5D synthetic functions. Performance is measured with a normalized regret score where higher is better.
• Figure 2: Mean normalized performance with standard error over 5 splits on holdout training distribution test functions from 2D to 10D. We test over 10 functions of each type from each dimension, totaling 900 overall functions.
• Figure 3: Mean normalized performance with standard error over 5 splits on out-of-distribution BBOB 2D to 10D test functions. We test over 50 functions of each type from each dimension, totaling 450 overall functions.
• Figure 4: Mean normalized performance with standard error over 5 splits on out-of-distribution VLSE 2D to 10D test functions. We test over 50 functions of each type from each dimension, totaling 450 overall functions.
• Figure 5: Ablation studies on training dataset and inference method.