LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning

TL;DR

The paper investigates whether LLMs can complement gradient-based optimization by acting as a prompt optimizer, proposing MaaO, a collaborative framework that alternates gradient-based prompt tuning with LLM-driven prompt generation using optimization trajectories as context. It formalizes prompt tuning with $\boldsymbol{\theta}$ and the gradient update $\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} - \eta_t \nabla_{\boldsymbol{\theta}} \mathcal{L}$, and introduces a token-space projection via $\mathcal{V}^{-1}$ to translate continuous prompts into discrete tokens for LLM input. Experiments on language and vision-language prompting tasks show consistent improvements over strong baselines, with GPT-4-based MaaO often delivering the largest gains. The work demonstrates that integrating LLM inference with conventional optimization can yield practical performance benefits and suggests broader applicability of LLM-driven guidance in optimization workflows, accompanied by code release.

Abstract

Mastering a skill generally relies on both hands-on experience from doers and insightful, high-level guidance by mentors. Will this strategy also work well for solving complex non-convex optimization problems? Here, a common gradient-based optimizer acts like a disciplined doer, making locally optimal updates at each step. Large Language Models (LLMs) can also search for better solutions by inferring from natural language instructions, akin to a high-level mentor. In this paper, we show that these two participators are complementary to each other and can effectively collaborate as a combined optimization framework. The collaborative optimization is achieved by alternating between the gradient-based and LLM-based optimizers. We instruct LLMs to generate possibly improved solutions by taking parameter trajectories recorded during the previous stage of gradient-based optimization into account. Inferred results of LLMs are used as restarting points for the next stage of gradient optimization. We verify the effectiveness of this optimization framework on prompt tuning. By leveraging both the locally rigorous gradient-based optimizer and the high-level deductive LLM-based optimizer, the combined optimization method consistently yields improvements over competitive baselines on a variety of tasks. Our results demonstrate the synergistic effect of conventional gradient-based optimization and the inference ability of LLMs. The code is released at https://github.com/guozix/LLM-catalyst.

Figures (6)

• Figure 1: Overview of our proposed method. The bold arrows with different color show the two collaborative optimizers of in our method. The thin arrows show the workflow of MaaO which infer for promising candidate prompt in vocabulary space for gradient-based optimizer.
• Figure 2: The result of gradient-based prompt optimization with different prompt initialization. The shadow denotes the standard deviation of the accuracy over three random seeds.
• Figure 3: The instruction used to query GPT-3.5 and GPT-4.0 in an iteration of optimizing the prompt using LLM.
• Figure 4: Interpretation of prompts optimized by LLM on EuroSAT dataset.
• Figure 5: The instruction used to query Llama2-7B-chat in an iteration of optimizing the prompt using LLM.