Towards Optimizing with Large Language Models

TL;DR

The paper investigates whether large language models can perform optimization tasks through interactive, iterative prompting across gradient-based, meta-heuristic, grid-search, and black-box settings. It introduces three metrics (Goal, Policy, Uncertainty) to assess performance across problem dimensions and task types. Core findings show strong optimization in small-scale problems, with performance influenced by data size and value ranges, and reveal potential for LLMs as hybrid optimizers alongside notable limitations. The work motivates further research on prompting strategies, stability techniques like self-consistency, and integration with classical optimization frameworks.

Abstract

In this work, we conduct an assessment of the optimization capabilities of LLMs across various tasks and data sizes. Each of these tasks corresponds to unique optimization domains, and LLMs are required to execute these tasks with interactive prompting. That is, in each optimization step, the LLM generates new solutions from the past generated solutions with their values, and then the new solutions are evaluated and considered in the next optimization step. Additionally, we introduce three distinct metrics for a comprehensive assessment of task performance from various perspectives. These metrics offer the advantage of being applicable for evaluating LLM performance across a broad spectrum of optimization tasks and are less sensitive to variations in test samples. By applying these metrics, we observe that LLMs exhibit strong optimization capabilities when dealing with small-sized samples. However, their performance is significantly influenced by factors like data size and values, underscoring the importance of further research in the domain of optimization tasks for LLMs.

Figures (17)

• Figure 1: Overview of our prompting strategy. (1) LLMs formulate the loss function based on given samples. (2) Given algorithm instructions and past results, LLM generates a new solution. (3) Calculate the loss of the new solution and add the solution-score pairs to the prompt of the next iteration. (4) Repeat the second and third steps until stop criteria are met.
• Figure 2: Goal Metric and Policy Metric hover from positive to near zero, signifying substantial optimization capability and alignment between LLM’s output and ground truth.
• Figure 3: Goal Metric reflects the performance of LLMs as Black-Box optimizer, showing strong performance with instances of smaller dimensions.
• Figure 4: Low values in the Policy Metric and high positive values in the Goal Metric indicate the robust performance of the LLM in the gradient descent task.
• Figure 5: An initial rise followed by a decline in the Uncertainty Metric with instance dimension growth suggests LLMs may have a richer sample space for small-scale problems, consistent across tasks and models.