Position: Leverage Foundational Models for Black-Box Optimization

TL;DR

This paper addresses the challenge of scaling black-box optimization (BBO) by leveraging foundational sequence-based models. It frames the BBO loop as: at iteration $t$, the optimizer proposes $x_t \in \mathcal{X}$, receives $y_t=f(x_t)$, and appends $(x_t,y_t)$ to the history $h_{s:t}$, with $\mathcal{H}$ denoting trajectory space. It argues that Transformer-based models can learn priors from world knowledge and historical evaluations, enabling data-driven, transferable optimization across heterogeneous search spaces and feedback types. The paper surveys prior approaches, outlines a taxonomy of search-space invariants, and identifies core challenges (data representation, multimodality, benchmarking) and future directions toward multi-modal, long-context BBO with LLMs. This vision promises more generalizable, scalable, and interactive optimization for experimental design across domains.

Abstract

Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave of innovation in the machine learning research domain, resulting in substantial impact across diverse fields such as reinforcement learning, robotics, and computer vision. Their incorporation has been rapid and transformative, marking a significant paradigm shift in the field of machine learning research. However, the field of experimental design, grounded on black-box optimization, has been much less affected by such a paradigm shift, even though integrating LLMs with optimization presents a unique landscape ripe for exploration. In this position paper, we frame the field of black-box optimization around sequence-based foundation models and organize their relationship with previous literature. We discuss the most promising ways foundational language models can revolutionize optimization, which include harnessing the vast wealth of information encapsulated in free-form text to enrich task comprehension, utilizing highly flexible sequence models such as Transformers to engineer superior optimization strategies, and enhancing performance prediction over previously unseen search spaces.

Figures (3)

• Figure 1: Foundation Models can learn priors from a wide variety of sources, such as world knowledge, domain-specific documents, and actual experimental evaluations. Such models can then perform black-box optimization over various search spaces (e.g. hyperparameters, code, natural language) and feedbacks (numeric values, categorical ratings, and subjective sentiment.
• Figure 2: Black-box optimization loop with sequential foundation models. Using metadata $m$ and history $h$, the model proposes candidates $x$ which are checked for feasibility, evaluated, and then appended to the history.
• Figure 3: Summary of future challenges and open questions for BBO with LLMs.