Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks

Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, Yuqing Yang

TL;DR

MLCopilot proposes an offline-online framework that leverages large language models to solve novel ML tasks by reusing past experiences. It canonicalizes heterogeneous historical data, elicits knowledge from them, and then retrieves relevant demonstrations and knowledge at inference time to prompt an LLM for a final solution. The approach yields competitive results on multiple ML benchmarks and demonstrates the value of retrieval-augmented reasoning with explicit knowledge elicitation. This work suggests a practical path to interpretable, rapid ML task solving that complements traditional AutoML methods and enables knowledge sharing across tasks.

Abstract

The field of machine learning (ML) has gained widespread adoption, leading to significant demand for adapting ML to specific scenarios, which is yet expensive and non-trivial. The predominant approaches towards the automation of solving ML tasks (e.g., AutoML) are often time-consuming and hard to understand for human developers. In contrast, though human engineers have the incredible ability to understand tasks and reason about solutions, their experience and knowledge are often sparse and difficult to utilize by quantitative approaches. In this paper, we aim to bridge the gap between machine intelligence and human knowledge by introducing a novel framework, which leverages the state-of-the-art large language models to develop ML solutions for novel tasks. We showcase the possibility of extending the capability of LLMs to comprehend structured inputs and perform thorough reasoning for solving novel ML tasks. And we find that, after some dedicated design, the LLM can (i) observe from the existing experiences of ML tasks and (ii) reason effectively to deliver promising results for new tasks. The solution generated can be used directly to achieve high levels of competitiveness. Examples and code available at https://github.com/microsoft/CoML.

MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks

TL;DR

MLCopilot proposes an offline-online framework that leverages large language models to solve novel ML tasks by reusing past experiences. It canonicalizes heterogeneous historical data, elicits knowledge from them, and then retrieves relevant demonstrations and knowledge at inference time to prompt an LLM for a final solution. The approach yields competitive results on multiple ML benchmarks and demonstrates the value of retrieval-augmented reasoning with explicit knowledge elicitation. This work suggests a practical path to interpretable, rapid ML task solving that complements traditional AutoML methods and enables knowledge sharing across tasks.

Abstract

The field of machine learning (ML) has gained widespread adoption, leading to significant demand for adapting ML to specific scenarios, which is yet expensive and non-trivial. The predominant approaches towards the automation of solving ML tasks (e.g., AutoML) are often time-consuming and hard to understand for human developers. In contrast, though human engineers have the incredible ability to understand tasks and reason about solutions, their experience and knowledge are often sparse and difficult to utilize by quantitative approaches. In this paper, we aim to bridge the gap between machine intelligence and human knowledge by introducing a novel framework, which leverages the state-of-the-art large language models to develop ML solutions for novel tasks. We showcase the possibility of extending the capability of LLMs to comprehend structured inputs and perform thorough reasoning for solving novel ML tasks. And we find that, after some dedicated design, the LLM can (i) observe from the existing experiences of ML tasks and (ii) reason effectively to deliver promising results for new tasks. The solution generated can be used directly to achieve high levels of competitiveness. Examples and code available at https://github.com/microsoft/CoML.
Paper Structure (77 sections, 1 equation, 3 figures, 12 tables, 2 algorithms)

This paper contains 77 sections, 1 equation, 3 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models
  4. Machine Learning and AutoML
  5. Preliminaries
  6. MLCopilot
  7. Overall Framework
  8. Offline Stage: Understanding and Reasoning
  9. Online Stage: Retrieving and Solving
  10. Task Description in Natural Language
  11. Retrieval
  12. Canonicalization
  13. Knowledge Elicitation
  14. Experiment
  15. Experiment Setup
  16. ...and 62 more sections

Figures (3)

  • Figure 1: Overview of MLCopilot. MLCopilot has offline and online stages. During the offline stage, it creates pools of experience and knowledge. In the online stage, it retrieves experience and knowledge based on the novel task description. Finally, MLCopilot invokes LLM and returns solutions.
  • Figure 2: Offline stage: canonicalization, knowledge elicitation.
  • Figure 3: Effect of the experience number and the number of solutions demonstrated for each task.