Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settings

TL;DR

This work tackles nonmyopic Bayesian optimization under dynamic, history-dependent costs by introducing LookaHES, which combines a multi-step $H$-Entropy Search framework with pathwise sampling and neural policy optimization to enable long-horizon planning beyond 20 steps. A neural policy amortizes decision variables across lookahead steps, while pathwise sampling curbs trajectory complexity, making scalable planning feasible in large, structured action spaces such as protein sequence edits. The method formalizes dynamic costs via Markovian and non-Markovian models, optimizes an EHIG-based objective with horizon $L$, and leverages autoregressive policies (including LLMs) to predict subsequent queries. Empirical results across nine synthetic benchmarks and real-world tasks (geospatial NASA night-light optimization and protein design) show LookaHES outperforming strong baselines, with notable gains in both continuous and discrete domains. The work provides a general, scalable, and cost-aware approach for robust long-horizon optimization in complex decision spaces, with practical implications for ML, statistics, and applied domains.

Abstract

Bayesian optimization (BO) is a common framework for optimizing black-box functions, yet most existing methods assume static query costs and rely on myopic acquisition strategies. We introduce LookaHES, a nonmyopic BO framework designed for dynamic, history-dependent cost environments, where evaluation costs vary with prior actions, such as travel distance in spatial tasks or edit distance in sequence design. LookaHES combines a multi-step variant of $H$-Entropy Search with pathwise sampling and neural policy optimization, enabling long-horizon planning beyond twenty steps without the exponential complexity of existing nonmyopic methods. The key innovation is the integration of neural policies, including large language models, to effectively navigate structured, combinatorial action spaces such as protein sequences. These policies amortize lookahead planning and can be integrated with domain-specific constraints during rollout. Empirically, LookaHES outperforms strong myopic and nonmyopic baselines across nine synthetic benchmarks from two to eight dimensions and two real-world tasks: geospatial optimization using NASA night-light imagery and protein sequence design with constrained token-level edits. In short, LookaHES provides a general, scalable, and cost-aware solution for robust long-horizon optimization in complex decision spaces, which makes it a useful tool for researchers in machine learning, statistics, and applied domains. Our implementation is available at https://github.com/sangttruong/nonmyopia.

Figures (19)

• Figure 1: Comparison of myopic and nonmyopic BO in a dynamic cost setting. The unshaded region denotes the current receptive field, i.e., the subset of the input space considered for the next query. Myopic BO (left) has a narrow receptive field, focusing only on immediate reward. Nonmyopic BO (right) expands the receptive field by accounting for future queries, enabling selection of points that may appear suboptimal in the short term but lead to higher long-term rewards (see acquisition values in the bottom row). This broader planning horizon allows the decision-maker to access high-reward regions that greedy strategies cannot reach. See Section \ref{['sec:dynamic cost setting']} for details.
• Figure 2: Process illustration. The black-box functions define complex objective landscapes in vector or protein spaces. The agent leverages a surrogate model to approximate the black-box function. The neural network policy generates the next queries to explore and exploit the optimization space. The query generation is guided by a dynamic cost function, ensuring efficient and targeted navigation of the search landscape.
• Figure 3: Queries across BO iterations with $\sigma=0.05$ and $r$-spotlight cost. Yellow and green points indicate the initial position and final action, respectively. LookaHES reaches the global optimum, whereas the others tend to be trapped in local optima.
• Figure 4: Final observed value at $\sigma=0.05$. From the north of each plot, counter-clockwise: Ackley, Ackley4D, Alpine, Cosine8, Hartmann, HolderTable, Levy, StyblinskiTang, SynGP. LookaHES consistently found global optimum across various cost structures.
• Figure 5: Fluorescence distribution by edit distance (top), observed fluorescence across BO steps (bottom left), and regret across BO steps (bottom right). Myopic methods are trapped in local minima ($\approx 1.5$ fluorescence), while our 12-step LookaHES anticipates the global maximum, achieving $\approx 2.7$ fluorescence.