Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representation

Xingyu Wu, Yan Zhong, Jibin Wu, Bingbing Jiang, Kay Chen Tan

TL;DR

The paper tackles the challenge of algorithm selection by introducing AS-LLM, a framework that uses pretrained Large Language Models to extract rich, high-dimensional algorithm features from code or descriptive text and pairs them with problem representations via separate encoders. A feature-selection module and a cosine-based fusion/decision stage enable robust matching between problems and candidate algorithms, addressing the bidirectional nature of the problem-algorithm relationship. The authors derive a theoretical upper bound on model complexity through an inductive Rademacher complexity analysis and demonstrate empirical gains on the ASlib benchmark, with ablations confirming the value of algorithm features and feature selection. The work suggests that leveraging algorithm features via LLMs can improve generalization, particularly with larger algorithm portfolios, while also outlining data-quality and scalability considerations for real-world deployment.

Abstract

Algorithm selection, a critical process of automated machine learning, aims to identify the most suitable algorithm for solving a specific problem prior to execution. Mainstream algorithm selection techniques heavily rely on problem features, while the role of algorithm features remains largely unexplored. Due to the intrinsic complexity of algorithms, effective methods for universally extracting algorithm information are lacking. This paper takes a significant step towards bridging this gap by introducing Large Language Models (LLMs) into algorithm selection for the first time. By comprehending the code text, LLM not only captures the structural and semantic aspects of the algorithm, but also demonstrates contextual awareness and library function understanding. The high-dimensional algorithm representation extracted by LLM, after undergoing a feature selection module, is combined with the problem representation and passed to the similarity calculation module. The selected algorithm is determined by the matching degree between a given problem and different algorithms. Extensive experiments validate the performance superiority of the proposed model and the efficacy of each key module. Furthermore, we present a theoretical upper bound on model complexity, showcasing the influence of algorithm representation and feature selection modules. This provides valuable theoretical guidance for the practical implementation of our method.

Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representation

TL;DR

The paper tackles the challenge of algorithm selection by introducing AS-LLM, a framework that uses pretrained Large Language Models to extract rich, high-dimensional algorithm features from code or descriptive text and pairs them with problem representations via separate encoders. A feature-selection module and a cosine-based fusion/decision stage enable robust matching between problems and candidate algorithms, addressing the bidirectional nature of the problem-algorithm relationship. The authors derive a theoretical upper bound on model complexity through an inductive Rademacher complexity analysis and demonstrate empirical gains on the ASlib benchmark, with ablations confirming the value of algorithm features and feature selection. The work suggests that leveraging algorithm features via LLMs can improve generalization, particularly with larger algorithm portfolios, while also outlining data-quality and scalability considerations for real-world deployment.

Abstract

Algorithm selection, a critical process of automated machine learning, aims to identify the most suitable algorithm for solving a specific problem prior to execution. Mainstream algorithm selection techniques heavily rely on problem features, while the role of algorithm features remains largely unexplored. Due to the intrinsic complexity of algorithms, effective methods for universally extracting algorithm information are lacking. This paper takes a significant step towards bridging this gap by introducing Large Language Models (LLMs) into algorithm selection for the first time. By comprehending the code text, LLM not only captures the structural and semantic aspects of the algorithm, but also demonstrates contextual awareness and library function understanding. The high-dimensional algorithm representation extracted by LLM, after undergoing a feature selection module, is combined with the problem representation and passed to the similarity calculation module. The selected algorithm is determined by the matching degree between a given problem and different algorithms. Extensive experiments validate the performance superiority of the proposed model and the efficacy of each key module. Furthermore, we present a theoretical upper bound on model complexity, showcasing the influence of algorithm representation and feature selection modules. This provides valuable theoretical guidance for the practical implementation of our method.
Paper Structure (16 sections, 1 theorem, 13 equations, 2 figures, 2 tables)

This paper contains 16 sections, 1 theorem, 13 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Algorithm Selection
  4. Difference between Feature-based and Feature-free Algorithm Selection:
  5. Algorithm Feature-based Algorithm Selection
  6. Difference between Probing Features and Algorithm Features:
  7. Algorithm Selection Based on LLMs
  8. Upper Bound of the Model Complexity
  9. Experiments
  10. Experiment Settings
  11. Dataset Description:
  12. Comparing Approaches:
  13. Evaluation metrics:
  14. Performance Comparison
  15. Ablation Study
  16. ...and 1 more sections

Key Result

Theorem 1

Let $|\mathcal{S}_{\mathcal{P}}|$ and $|\mathcal{S}_{\mathcal{A}}|$ denote the number of problems and algorithms in training samples, $W^{(i)}$ denote the parameter in the $i$-th layer with the upper bound of $F$-norm $R_i$, i.e., $\|W^{(i)}\|_F\leq R_i$, and $l$ denote the number of layers. Then, f where $\Gamma_{\mathbf{f}}$ and $\Gamma_{\mathcal{S}}$ are model-related and data-related variables

Figures (2)

  • Figure 1: The framework of AS-LLM: (a) illustrates the various types of information that influence algorithm representation. (b) shows the algorithm features extracted by LLM and their preprocessing using feature selection. (c) outlines the proposed algorithm selection framework.
  • Figure 2: Ablation study on ASlib benchmarks.

Theorems & Definitions (2)

  • Theorem 1
  • proof