PLHF: Prompt Optimization with Few-Shot Human Feedback

TL;DR

PLHF tackles prompt optimization for generative LLMs under the absence of a clear metric and limited labeled data. It introduces a duo-module design with Responder $R$ and Evaluator $E$ where $E$ mimics human judgments by learning prompts $P_E$ via a simple metric $L$, and $R$ is optimized with $E$ as the reward, producing updated prompts $P'_R$ and $P'_E$ in alternating rounds. The method achieves superior output quality across public datasets SGD, AES-ASAP, AES-2.0, and an industrial SQL-QA task, outperforming string-based graders and standard PO baselines. The approach reduces human labeling cost by requiring at most a linear number of human feedback calls in the training set, and demonstrates practical applicability of few-shot, RLHF-inspired prompting.

Abstract

Automatic prompt optimization frameworks are developed to obtain suitable prompts for large language models (LLMs) with respect to desired output quality metrics. Although existing approaches can handle conventional tasks such as fixed-solution question answering, defining the metric becomes complicated when the output quality cannot be easily assessed by comparisons with standard golden samples. Consequently, optimizing the prompts effectively and efficiently without a clear metric becomes a critical challenge. To address the issue, we present PLHF (which stands for "P"rompt "L"earning with "H"uman "F"eedback), a few-shot prompt optimization framework inspired by the well-known RLHF technique. Different from naive strategies, PLHF employs a specific evaluator module acting as the metric to estimate the output quality. PLHF requires only a single round of human feedback to complete the entire prompt optimization process. Empirical results on both public and industrial datasets show that PLHF outperforms prior output grading strategies for LLM prompt optimizations.

Figures (4)

• Figure 1: Demonstrations of the actual failure cases that the evaluations from pre-trained LLMs have different preference from specific humans. The first (left) example is the task of joke generation, where the grading is according to funniness and novelty. The second scenario is a math problem generation bot, where the response quality is evaluated based on helpfulness and problem quality. As shown above, the verdicts of state-of-the-art LLMs could still differ from real human's preferences.
• Figure 2: Workflow framework of PLHF. The entire LLM program contains two modules, Responder $R$ and Evaluator $E$, where PO can be PO arbitrary prompt optimization method.
• Figure 3: A toy example to illustrate the subtask designs of PLHF. The targeted generative AI task for this example is "generate the punchline for a joke setup." The training samples $D$ are triplets (Input, Output, Labeled Score), where Input is the joke setup, Output is a sample output of the punchline for the corresponding Input, and Labeled Score is the rating judged by human experts. For this example, we consider Labeled Score$\geq 8$ as the condition of positive samples. Examples of optimized prompts $P'_E$ and $P'_R$ (for the evaluator $E$ and the responder $R$, respectively) are shown.
• Figure 4: Performance curves of PLHF on the datasets SGD and AES-ASAP. For the plots, we consider DSPy as the PO method for PLHF. The $x$-values are the number of (randomly selected) training samples. The $y$-values are mean values of the RMSE losses for $E$ and the output scores for $R$, respectively. The vertical bar of each point indicates the standard deviations estimated in 30 runs.