Prompt Optimization with Human Feedback

TL;DR

This work introduces POHF and APOHF for optimizing prompts to black-box LLMs using only human preferences, eliminating the need for numeric scores. APOHF learns a latent score $u(x)$ from prompt embeddings with a neural predictor and selects prompt pairs through a greedy-first and upper-confidence-bound second-arm strategy inspired by dueling bandits. Across instruction optimization, text-to-image prompting, and response refinement, APOHF demonstrates sample-efficient discovery of high-quality prompts, outperforming Random, Linear Dueling Bandits, and DoubleTS baselines. The approach is validated via multiple experiments and ablations, with discussions of limitations and potential societal impacts. Code is available at the authors’ GitHub.

Abstract

Large language models (LLMs) have demonstrated remarkable performances in various tasks. However, the performance of LLMs heavily depends on the input prompt, which has given rise to a number of recent works on prompt optimization. However, previous works often require the availability of a numeric score to assess the quality of every prompt. Unfortunately, when a human user interacts with a black-box LLM, attaining such a score is often infeasible and unreliable. Instead, it is usually significantly easier and more reliable to obtain preference feedback from a human user, i.e., showing the user the responses generated from a pair of prompts and asking the user which one is preferred. Therefore, in this paper, we study the problem of prompt optimization with human feedback (POHF), in which we aim to optimize the prompt for a black-box LLM using only human preference feedback. Drawing inspiration from dueling bandits, we design a theoretically principled strategy to select a pair of prompts to query for preference feedback in every iteration, and hence introduce our algorithm named automated POHF (APOHF). We apply our APOHF algorithm to various tasks, including optimizing user instructions, prompt optimization for text-to-image generative models, and response optimization with human feedback (i.e., further refining the response using a variant of our APOHF). The results demonstrate that our APOHF can efficiently find a good prompt using a small number of preference feedback instances. Our code can be found at \url{https://github.com/xqlin98/APOHF}.

Figures (9)

• Figure 1: Illustration of our automated prompt optimization with human feedback (APOHF).
• Figure 2: Latent scores of different methods in user instruction optimization, averaged over 30 tasks (Sec. \ref{['subsec:exp:instruction']}).
• Figure 3: Performances in prompt optimization for image generation in Sec. \ref{['subsec:exp:image']} (4 different scenes).
• Figure 4: Images generated by the best prompt discovered by our APOHF across different iterations.
• Figure 5: Scores of different methods for response optimization (Sec. \ref{['subsec:exp:response:opt']}).