Meta-Black-Box-Optimization through Offline Q-function Learning

TL;DR

MetaBBO faces efficiency and generalization challenges when learning DAC policies online. Q-Mamba answers with offline Q-learning that decomposes the high-dimensional configuration space into per-dimension Q-functions and employs a Mamba-based long-sequence learner, trained on a mixed offline dataset with a conservative Q-learning loss. The results show Q-Mamba achieving competitive or superior optimization performance while reducing training cost, and demonstrating zero-shot generalization to realistic neuroevolution tasks. This offline approach offers a practical pathway to scalable, data-efficient meta-learning for dynamic algorithm configuration in BBO problems.

Abstract

Recent progress in Meta-Black-Box-Optimization (MetaBBO) has demonstrated that using RL to learn a meta-level policy for dynamic algorithm configuration (DAC) over an optimization task distribution could significantly enhance the performance of the low-level BBO algorithm. However, the online learning paradigms in existing works makes the efficiency of MetaBBO problematic. To address this, we propose an offline learning-based MetaBBO framework in this paper, termed Q-Mamba, to attain both effectiveness and efficiency in MetaBBO. Specifically, we first transform DAC task into long-sequence decision process. This allows us further introduce an effective Q-function decomposition mechanism to reduce the learning difficulty within the intricate algorithm configuration space. Under this setting, we propose three novel designs to meta-learn DAC policy from offline data: we first propose a novel collection strategy for constructing offline DAC experiences dataset with balanced exploration and exploitation. We then establish a decomposition-based Q-loss that incorporates conservative Q-learning to promote stable offline learning from the offline dataset. To further improve the offline learning efficiency, we equip our work with a Mamba architecture which helps long-sequence learning effectiveness and efficiency by selective state model and hardware-aware parallel scan respectively. Through extensive benchmarking, we observe that Q-Mamba achieves competitive or even superior performance to prior online/offline baselines, while significantly improving the training efficiency of existing online baselines. We provide sourcecodes of Q-Mamba at https://github.com/MetaEvo/Q-Mamba.

Figures (5)

• Figure 1: The workflow of the Mamba-based Q-Learner. The forward process of the neural network is similar with the Recurrent Neural Network. At each time step, the Q-function of each decomposed action dimension is outputted by conditioning the current state and selected action bins of the previous action dimensions. The environment transition is executed once all action dimensions are outputted.
• Figure 2: Zero shot performance of Q-Mamba and online MetaBBO baselines on neuroevolution tasks.
• Figure 3: Fitness landscapes of functions in BBOB train set when dimension is set to 2.
• Figure 4: Fitness landscapes of functions in BBOB test set when dimension is set to 2.
• Figure 5: The performances of Q-Mamba trained with different action bin granularities.