Prompt Optimization with Logged Bandit Data

TL;DR

The paper tackles optimizing prompt policies for language generation using naturally logged user feedback by formulating the task as off-policy learning with contextual bandits and auxiliary sentence outputs. It introduces Direct Sentence Off-policy Gradient (DSO), a kernel-based method that performs gradient estimation in the sentence space via marginalized utilities, significantly reducing variance and bias compared with traditional action-space IS or regression approaches. The authors provide a theoretical bias-variance analysis under a Similar Sentence Support condition and validate the approach on both synthetic benchmarks and a full-LLM MovieLens-based task using OfflinePrompts, achieving up to a 5x improvement in challenging, large-prompt scenarios. The work also contributes an open-source benchmark suite that facilitates reproducible OPL research for prompt tuning and demonstrates practical impact for personalized language generation applications.

Abstract

We study how to use naturally available user feedback, such as clicks, to optimize large language model (LLM) pipelines for generating personalized sentences using prompts. Naive approaches, which estimate the policy gradient in the prompt space, suffer either from variance caused by the large action space of prompts or bias caused by inaccurate reward predictions. To circumvent these challenges, we propose a novel kernel-based off-policy gradient method, which estimates the policy gradient by leveraging similarity among generated sentences, substantially reducing variance while suppressing the bias. Empirical results on our newly established suite of benchmarks demonstrate the effectiveness of the proposed approach in generating personalized descriptions for movie recommendations, particularly when the number of candidate prompts is large.

Figures (25)

• Figure 1: Overview of the prompt-based sentence personalization with logged bandit feedback. For each coming user, a policy chooses which prompt to use to generate sentences with a frozen LLM. Each user observes only the sentence generated by the chosen prompt and provides the reward for the corresponding sentence. Logged bandit feedback is partial in that we cannot observe rewards for the sentences generated by prompts not chosen by the logging policy. Examples are generated by ChatGPT-3.5 brown2020language.
• Figure 2: Examples of the kernel weights and (soft) rejection sampling in the marginalized sentence space. DSO implicitly augments the data to take the observations for the neighboring sentences into account. (Left) uses a smooth kernel like a Gaussian kernel, and (Right) uses a piecewise constant kernel like a uniform kernel.
• Figure 3: Bias-variance tradeoff of DSO and its relations to the bandwidth hyperparameter ($\tau$) of a kernel function: When $\tau$ is large, the overlap between the logging policy ($\pi_0$) and the current policy ($\pi_{\theta}$) within $\phi(s)$ becomes large, thus the scale of the importance weight becomes small. This contributes to reducing the variance compared to naive IS. In contrast, a small value of $\tau$ helps keep the bias small, as the within-neighbor reward shift (i.e., the difference between $q^{\pi_0}(x,\phi(s))$ and $q^{\pi_{\theta}}(x,\phi(s))$) becomes small. The gray regions are rejected when using a uniform kernel.
• Figure 4: Comparing the performance of the policies learned by various OPL methods with (Left) varying data sizes ($n$), (Middle) varying number of candidate actions ($|\mathcal{A}|$), and (Right) varying reward noises ($\sigma_r$). DSO uses a Gaussian kernel and function approximation of $\pi_0(\phi(s)|x)$.
• Figure 5: Ablation results of DSO with varying bandwidth hyperparameters ($\tau$), w/ and w/o function approximation of $\pi_0(\phi(s)|x)$, and two kernels, Gaussian and uniform.

Theorems & Definitions (5)

• Definition 1
• Theorem 1
• Theorem 2
• proof
• proof