Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks

Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theodorakopoulos, Tanja Tornede, Henning Wachsmuth, Marius Lindauer

TL;DR

The paper investigates the intersection of Automated Machine Learning (AutoML) and Large Language Models (LLMs), arguing that a tight, symbiotic integration can advance both NLP and AutoML. It reviews current work and organizes the challenges of applying AutoML to LLMs (costly pre-training, multi-stage lifecycles, diverse performance metrics, multi-paradigm optimization, and NAS) while outlining opportunities for LLMs to enhance AutoML through human-machine interfaces, problem configuration, and embedded components. The authors present a structured critique of risks—ranging from complicated human interaction and evaluation bias to hallucinations and escalating resource use—and highlight the need for careful, transparent, and potentially open-source development to responsibly harness these technologies. Overall, the work maps a roadmap for advancing AutoML-informed LLM development and LLM-enabled AutoML tooling, stressing pragmatic approaches and community-driven solutions to manage resource constraints and trust concerns.

Abstract

The fields of both Natural Language Processing (NLP) and Automated Machine Learning (AutoML) have achieved remarkable results over the past years. In NLP, especially Large Language Models (LLMs) have experienced a rapid series of breakthroughs very recently. We envision that the two fields can radically push the boundaries of each other through tight integration. To showcase this vision, we explore the potential of a symbiotic relationship between AutoML and LLMs, shedding light on how they can benefit each other. In particular, we investigate both the opportunities to enhance AutoML approaches with LLMs from different perspectives and the challenges of leveraging AutoML to further improve LLMs. To this end, we survey existing work, and we critically assess risks. We strongly believe that the integration of the two fields has the potential to disrupt both fields, NLP and AutoML. By highlighting conceivable synergies, but also risks, we aim to foster further exploration at the intersection of AutoML and LLMs.

AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks

TL;DR

The paper investigates the intersection of Automated Machine Learning (AutoML) and Large Language Models (LLMs), arguing that a tight, symbiotic integration can advance both NLP and AutoML. It reviews current work and organizes the challenges of applying AutoML to LLMs (costly pre-training, multi-stage lifecycles, diverse performance metrics, multi-paradigm optimization, and NAS) while outlining opportunities for LLMs to enhance AutoML through human-machine interfaces, problem configuration, and embedded components. The authors present a structured critique of risks—ranging from complicated human interaction and evaluation bias to hallucinations and escalating resource use—and highlight the need for careful, transparent, and potentially open-source development to responsibly harness these technologies. Overall, the work maps a roadmap for advancing AutoML-informed LLM development and LLM-enabled AutoML tooling, stressing pragmatic approaches and community-driven solutions to manage resource constraints and trust concerns.

Abstract

The fields of both Natural Language Processing (NLP) and Automated Machine Learning (AutoML) have achieved remarkable results over the past years. In NLP, especially Large Language Models (LLMs) have experienced a rapid series of breakthroughs very recently. We envision that the two fields can radically push the boundaries of each other through tight integration. To showcase this vision, we explore the potential of a symbiotic relationship between AutoML and LLMs, shedding light on how they can benefit each other. In particular, we investigate both the opportunities to enhance AutoML approaches with LLMs from different perspectives and the challenges of leveraging AutoML to further improve LLMs. To this end, we survey existing work, and we critically assess risks. We strongly believe that the integration of the two fields has the potential to disrupt both fields, NLP and AutoML. By highlighting conceivable synergies, but also risks, we aim to foster further exploration at the intersection of AutoML and LLMs.
Paper Structure (31 sections, 3 figures)

This paper contains 31 sections, 3 figures.

Table of Contents

  1. Introduction
  2. AutoML for LLMs
  3. Background on LLMs
  4. Challenge I: Cost of Pre-Training Base Models
  5. Prior-Guided Multi-Fidelity Optimization with Scaling Laws
  6. Gradient-Based AutoML
  7. Meta-Learning When and How to Adjust Training
  8. Challenge II: A Multitude of Different Stages
  9. Pre-Training
  10. Fine-Tuning
  11. Supervised Task-Specific Fine-Tuning
  12. Instruction Tuning
  13. Alignment Fine-Tuning
  14. Alignment Fine-Tuning
  15. Inference
  16. ...and 16 more sections

Figures (3)

  • Figure 1: AutoML can be used in all stages of the LLM lifecycle and needs to be adjusted to the different objectives, hyperparameters, and design decisions of each stage. The graphic depicts exemplary objectives, subjects of optimization, and associated hyperparameters. Due to computational constraints, the stages are considered separately, one after the other.
  • Figure 2: Overview of options where LLMs can be integrated into the AutoML process.
  • Figure 3: Visualization of the potential of LLMs for the configuration of AutoML (\ref{['subsec:llms-for-configuring-automl']}) and LLMs as components of AutoML systems (\ref{['subsec:llms-for-simulating-automl']}) at the example of a AutoML process based on Bayesian Optimization.