Large Language Model as Meta-Surrogate for Data-Driven Many-Task Optimization: A Proof-of-Principle Study

TL;DR

The paper tackles the high cost of evaluating multiple optimization tasks by introducing an LLM-based meta-surrogate that treats fitness prediction as $p(y\mid x,m)$ and leverages tokenized inputs, including a Scientific Notation Encoding for numerics. It demonstrates that a single, offline-finetuned model can predict cross-task fitness across varying dimensions, showing emergent zero-shot generalization and competitive performance against traditional surrogates. The approach is integrated with offline data-driven MaTOP algorithms (MaTDE and BLKT-DE), yielding consistent improvements in multi-task optimization benchmarks. This work highlights a scalable, cross-task surrogate paradigm that exploits LLM priors for enhanced efficiency and robustness in data-driven many-task optimization.

Abstract

In many-task optimization scenarios, surrogate models are valuable for mitigating the computational burden of repeated fitness evaluations across tasks. This study proposes a novel meta-surrogate framework to assist many-task optimization, by leveraging the knowledge transfer strengths and emergent capabilities of large language models (LLMs). We formulate a unified framework for many-task fitness prediction, by defining a universal model with metadata to fit a group of problems. Fitness prediction is performed on metadata and decision variables, enabling efficient knowledge sharing across tasks and adaptability to new tasks. The LLM-based meta-surrogate treats fitness prediction as conditional probability estimation, employing a unified token sequence representation for task metadata, inputs, and outputs. This approach facilitates efficient inter-task knowledge sharing through shared token embeddings and captures complex task dependencies via multi-task model training. Experimental results demonstrate the model's emergent generalization ability, including zero-shot performance on problems with unseen dimensions. When integrated into evolutionary transfer optimization (ETO), our framework supports dual-level knowledge transfer -- at both the surrogate and individual levels -- enhancing optimization efficiency and robustness. This work establishes a novel foundation for applying LLMs in surrogate modeling, offering a versatile solution for many-task optimization.

Figures (4)

• Figure 1: Meta-Surrogate Framework
• Figure 2: Prediction of decision variables and objective values based on LLM
• Figure 3: Prediction Errors Associated with Different Sampling Methods.
• Figure 4: Comparison of the relative RMSE values of MetaSurrogate(FEs=0) and RBFN(FEs=30) with respect to RBFN(FEs=20)$=$1 across different dimensions.