Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey

TL;DR

This survey investigates how Large Language Models can be integrated to construct and optimize machine learning workflows, spanning data/feature engineering, model selection, hyperparameter optimization, and workflow evaluation. It details two broad LLM-enabled approaches for each stage—retrieval- and generation-based methods—and contrasts execution-based versus prediction-based HPO, highlighting methods like HPO-LLaMA, GPT-NAS, ModelGPT, and CAAFE. The authors discuss benefits such as reduced manual effort, interpretability via feature explanations, and faster evaluation through simulation, while candidly addressing challenges including data leakage, hallucinations, prompt design complexity, and resource demands. The work also outlines open problems and future directions, including end-to-end LLM-driven workflow construction and hybrid systems that couple LLMs with specialized models to improve reliability and practicality across domains.

Abstract

Building effective machine learning (ML) workflows to address complex tasks is a primary focus of the Automatic ML (AutoML) community and a critical step toward achieving artificial general intelligence (AGI). Recently, the integration of Large Language Models (LLMs) into ML workflows has shown great potential for automating and enhancing various stages of the ML pipeline. This survey provides a comprehensive and up-to-date review of recent advancements in using LLMs to construct and optimize ML workflows, focusing on key components encompassing data and feature engineering, model selection and hyperparameter optimization, and workflow evaluation. We discuss both the advantages and limitations of LLM-driven approaches, emphasizing their capacity to streamline and enhance ML workflow modeling process through language understanding, reasoning, interaction, and generation. Finally, we highlight open challenges and propose future research directions to advance the effective application of LLMs in ML workflows.

Figures (5)

• Figure 1: An overview of the Machine Learning Workflow, where task specification serves as the input, encompassing key stages of data and feature engineering, model selection and hyperparameter optimization, and workflow evaluation.
• Figure 2: The main categorization of LLM-assisted Data Preprocessing methods.
• Figure 3: An example of feature synthesis by LLMs. Refer to CAAFE hollmann2024large for detailed description.
• Figure 4: The illustration of two distinct approaches for LLM-assisted Model Selection methods.
• Figure 5: The overview of two categories of LLM-assisted Hyperparameter Optimization methods.