Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Fidelity Methods for Optimization: A Survey

Ke Li, Fan Li

TL;DR

The paper surveys multi-fidelity optimization (MFO) for expensive black-box problems, introducing a text-mining framework to map the literature and identify core components. It provides a structured analysis of three building blocks—surrogate modeling, fidelity management, and optimization techniques—alongside a taxonomy of MF research, benchmark issues, and domain applications (HPO, engineering design, scientific discovery, and PINNs). Key contributions include a comprehensive review of surrogate-model families (single-model, space-mapping, correction-based, AR1, MTGP, nonlinear hierarchical, and MF-PINNs), optimizer classes (BO, SAEAs, bandits), and fidelity strategies (fixed vs adaptive), plus critical discussions on benchmarks and reproducibility. The survey highlights challenges in scalability, cost–accuracy balancing, human-in-the-loop integration, near real-world benchmarks, and open science, offering guidance to drive future research and cross-disciplinary collaboration in MFO.

Abstract

Real-world black-box optimization often involves time-consuming or costly experiments and simulations. Multi-fidelity optimization (MFO) stands out as a cost-effective strategy that balances high-fidelity accuracy with computational efficiency through a hierarchical fidelity approach. This survey presents a systematic exploration of MFO, underpinned by a novel text mining framework based on a pre-trained language model. We delve deep into the foundational principles and methodologies of MFO, focusing on three core components -- multi-fidelity surrogate models, fidelity management strategies, and optimization techniques. Additionally, this survey highlights the diverse applications of MFO across several key domains, including machine learning, engineering design optimization, and scientific discovery, showcasing the adaptability and effectiveness of MFO in tackling complex computational challenges. Furthermore, we also envision several emerging challenges and prospects in the MFO landscape, spanning scalability, the composition of lower fidelities, and the integration of human-in-the-loop approaches at the algorithmic level. We also address critical issues related to benchmarking and the advancement of open science within the MFO community. Overall, this survey aims to catalyze further research and foster collaborations in MFO, setting the stage for future innovations and breakthroughs in the field.

Multi-Fidelity Methods for Optimization: A Survey

TL;DR

The paper surveys multi-fidelity optimization (MFO) for expensive black-box problems, introducing a text-mining framework to map the literature and identify core components. It provides a structured analysis of three building blocks—surrogate modeling, fidelity management, and optimization techniques—alongside a taxonomy of MF research, benchmark issues, and domain applications (HPO, engineering design, scientific discovery, and PINNs). Key contributions include a comprehensive review of surrogate-model families (single-model, space-mapping, correction-based, AR1, MTGP, nonlinear hierarchical, and MF-PINNs), optimizer classes (BO, SAEAs, bandits), and fidelity strategies (fixed vs adaptive), plus critical discussions on benchmarks and reproducibility. The survey highlights challenges in scalability, cost–accuracy balancing, human-in-the-loop integration, near real-world benchmarks, and open science, offering guidance to drive future research and cross-disciplinary collaboration in MFO.

Abstract

Real-world black-box optimization often involves time-consuming or costly experiments and simulations. Multi-fidelity optimization (MFO) stands out as a cost-effective strategy that balances high-fidelity accuracy with computational efficiency through a hierarchical fidelity approach. This survey presents a systematic exploration of MFO, underpinned by a novel text mining framework based on a pre-trained language model. We delve deep into the foundational principles and methodologies of MFO, focusing on three core components -- multi-fidelity surrogate models, fidelity management strategies, and optimization techniques. Additionally, this survey highlights the diverse applications of MFO across several key domains, including machine learning, engineering design optimization, and scientific discovery, showcasing the adaptability and effectiveness of MFO in tackling complex computational challenges. Furthermore, we also envision several emerging challenges and prospects in the MFO landscape, spanning scalability, the composition of lower fidelities, and the integration of human-in-the-loop approaches at the algorithmic level. We also address critical issues related to benchmarking and the advancement of open science within the MFO community. Overall, this survey aims to catalyze further research and foster collaborations in MFO, setting the stage for future innovations and breakthroughs in the field.
Paper Structure (52 sections, 1 equation, 7 figures)

This paper contains 52 sections, 1 equation, 7 figures.

Table of Contents

  1. Introduction
  2. Meta-analysis of multi-fidelity optimization literature: A text mining approach
  3. Text Mining Framework
  4. Text Mining Results
  5. Text mining from publications
  6. Text mining from topic modeling
  7. Taxonomy of MFO research
  8. Multi-Fidelity Surrogate Modeling
  9. Single Model Methods
  10. Space-Mapping Methods
  11. Correction-based Methods
  12. Autoregressive-based (AR1) Methods
  13. Multi-task Gaussian Process (MTGP) Methods
  14. Nonlinear Hierarchical Methods
  15. Multi-Fidelity Physics-Informed Neural Networks (PINNs)
  16. ...and 37 more sections

Figures (7)

  • Figure 1: The flowchart of the text mining framework underpinning our survey. It is a closed-loop process where human experts' feedback will be fed back into data collection, data cleaning, and topic modeling steps to improve the quality of text mining results.
  • Figure 2: The temporal statistics from 1998 to 2023 regarding: (a) the publication trend of ten most active publication venues (represented in their acronyms) , and (b) the progression of top ten most popular topics learned from the topic modeling.
  • Figure 3: Geographical distribution of publications across different regions. To minimize information cluttering, we only list at most top ten most productive regions in each continent.
  • Figure 4: Word clouds of $23$ topics learned by our topic modeling pipeline from the collected articles. Note that the topics are indexed according to a descending order of the number of underpinned articles.
  • Figure 5: A comparison among USA, China, and EU (including UK and Switzerland) about the temporal statistics from 1998 to 2023 regarding the progression of topics. To mitigate information cluttering, we only display the top ten most popular topics.
  • ...and 2 more figures