The Importance of Directional Feedback for LLM-based Optimizers

TL;DR

The paper investigates using large language models as general optimizers for problems expressed in natural language, introducing a formal distinction between directional and non-directional feedback and showing that directional signals enable more effective search in text space. It proposes Sequential Prompt Optimization, a framework where an LLM-based optimizer updates a tunable prompt $p_{\text{tunable}}$ based on a history of outputs, rewards, and feedback, aided by a Feedback Synthesizer and a Prompt Selector to ensure monotonic improvement. The authors provide theoretical arguments and empirical evidence across two domains—the optimization of mathematical functions and the crafting of poetry prompts—demonstrating that directional (or synthesized) feedback yields greater stability and efficiency than non-directional or reward-only feedback. The work suggests broad potential for LLM-based optimizers to tackle diverse, nontraditional optimization tasks, while calling for further methods to generate effective directional feedback in practice.

Abstract

We study the potential of using large language models (LLMs) as an interactive optimizer for solving maximization problems in a text space using natural language and numerical feedback. Inspired by the classical optimization literature, we classify the natural language feedback into directional and non-directional, where the former is a generalization of the first-order feedback to the natural language space. We find that LLMs are especially capable of optimization when they are provided with {directional feedback}. Based on this insight, we design a new LLM-based optimizer that synthesizes directional feedback from the historical optimization trace to achieve reliable improvement over iterations. Empirically, we show our LLM-based optimizer is more stable and efficient in solving optimization problems, from maximizing mathematical functions to optimizing prompts for writing poems, compared with existing techniques.

Figures (5)

• Figure 1: We visualize the optimization trajectory path made by the Optimizer Agent with GPT-3.5 and GPT-4. The loss landscape on the left is the Rosenbrock Function, and on the right is the Six-Hump Camel Function.
• Figure 2: We plot the average Cumulative Regret and Simple Regret of each condition over 10 trials. Each algorithm is allowed to take 10 steps. We tuned the SGD learning rate to ensure that it was not too large or too small. The result is aggregated over 4 loss functions.
• Figure 3: We plot the average Cumulative Regret and Simple Regret of each condition over 10 trials and compare different feedback types. Synthetic Feedback is generated by the same LLM as the optimizer.
• Figure 4: We show the reward for each policy after each round of interaction with the environment. OptAgent (our algorithm) is in red.
• Figure :