Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

READY: Reward Discovery for Meta-Black-Box Optimization

Zechuan Huang, Zhiguang Cao, Hongshu Guo, Yue-Jiao Gong, Zeyuan Ma

TL;DR

This work introduces READY, an LLM-driven, multitask reward discovery framework for MetaBBO. By combining a niche-based population, fine-grained evolutionary operators, and explicit knowledge transfer, READY autonomously designs ready-to-deploy rewards that improve meta-optimization performance across heterogeneous BBO agents. Empirical results on a BBOB-style multitask setup show READY outperforms handcrafted rewards and competitive automated baselines, with strong zero-shot generalization and improved stability. The approach demonstrates scalable, interpretable reward design that can accelerate the development of autonomous optimization systems with broad practical impact.

Abstract

Meta-Black-Box Optimization (MetaBBO) is an emerging avenue within Optimization community, where algorithm design policy could be meta-learned by reinforcement learning to enhance optimization performance. So far, the reward functions in existing MetaBBO works are designed by human experts, introducing certain design bias and risks of reward hacking. In this paper, we use Large Language Model~(LLM) as an automated reward discovery tool for MetaBBO. Specifically, we consider both effectiveness and efficiency sides. On effectiveness side, we borrow the idea of evolution of heuristics, introducing tailored evolution paradigm in the iterative LLM-based program search process, which ensures continuous improvement. On efficiency side, we additionally introduce multi-task evolution architecture to support parallel reward discovery for diverse MetaBBO approaches. Such parallel process also benefits from knowledge sharing across tasks to accelerate convergence. Empirical results demonstrate that the reward functions discovered by our approach could be helpful for boosting existing MetaBBO works, underscoring the importance of reward design in MetaBBO. We provide READY's project at https://anonymous.4open.science/r/ICML_READY-747F.

READY: Reward Discovery for Meta-Black-Box Optimization

TL;DR

This work introduces READY, an LLM-driven, multitask reward discovery framework for MetaBBO. By combining a niche-based population, fine-grained evolutionary operators, and explicit knowledge transfer, READY autonomously designs ready-to-deploy rewards that improve meta-optimization performance across heterogeneous BBO agents. Empirical results on a BBOB-style multitask setup show READY outperforms handcrafted rewards and competitive automated baselines, with strong zero-shot generalization and improved stability. The approach demonstrates scalable, interpretable reward design that can accelerate the development of autonomous optimization systems with broad practical impact.

Abstract

Meta-Black-Box Optimization (MetaBBO) is an emerging avenue within Optimization community, where algorithm design policy could be meta-learned by reinforcement learning to enhance optimization performance. So far, the reward functions in existing MetaBBO works are designed by human experts, introducing certain design bias and risks of reward hacking. In this paper, we use Large Language Model~(LLM) as an automated reward discovery tool for MetaBBO. Specifically, we consider both effectiveness and efficiency sides. On effectiveness side, we borrow the idea of evolution of heuristics, introducing tailored evolution paradigm in the iterative LLM-based program search process, which ensures continuous improvement. On efficiency side, we additionally introduce multi-task evolution architecture to support parallel reward discovery for diverse MetaBBO approaches. Such parallel process also benefits from knowledge sharing across tasks to accelerate convergence. Empirical results demonstrate that the reward functions discovered by our approach could be helpful for boosting existing MetaBBO works, underscoring the importance of reward design in MetaBBO. We provide READY's project at https://anonymous.4open.science/r/ICML_READY-747F.
Paper Structure (58 sections, 4 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 58 sections, 4 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. MetaBBO
  4. Automated Reward Discovery via LLMs
  5. Methodology
  6. Problem Formulation
  7. Initialization with Greedy Sampling
  8. Niche Specific Metadata Modeling
  9. Gene Encoding
  10. Greedy Niche-based Population Sampling
  11. Diversified Evolutionary Operators
  12. Population Management
  13. Knowledge Transfer
  14. Experiments
  15. Experimental Setup
  16. ...and 43 more sections

Figures (3)

  • Figure 1: General workflow of MetaBBO approaches.
  • Figure 2: Evolutionary trajectory of READY's best-so-far performance when discover reward for DEDQN. The plateau reflects a phase of strategic accumulation, serving as a necessary precursor for the significant exploitation breakthrough.
  • Figure 3: Ablation and sensitivity analysis. Y-axis shows Summed Normalized Efficiency, with the red dashed line ($y=1.0$) marking the full model baseline.