Dialogue for Prompting: a Policy-Gradient-Based Discrete Prompt Generation for Few-shot Learning

TL;DR

This paper tackles discrete prompt optimization for few-shot PLMs by introducing DP2O, a two-stage framework that first builds a high-quality, readable discrete prompt set via multi-round GPT-4 dialogue alignment guided by a novel Supervised & Unsupervised Entropy (SUE) metric, and then uses a policy-gradient RL agent to match prompts to inputs and ensemble their predictions. The method achieves state-of-the-art-like gains with only a small fraction of PLM parameters trained, demonstrating improved accuracy across four datasets and showing robustness and generalization across model sizes and tasks. The work provides a practical, efficient path to prompt-enabled few-shot learning with strong transferability and readability, addressing both effectiveness and usability in real-world applications. Overall, DP2O advances discrete prompt optimization by combining dialogue-driven prompt construction, a principled prompt quality metric, and reinforcement learning-based prompt matching to enable readable, scalable, and generalizable few-shot reasoning.

Abstract

Prompt-based pre-trained language models (PLMs) paradigm have succeeded substantially in few-shot natural language processing (NLP) tasks. However, prior discrete prompt optimization methods require expert knowledge to design the base prompt set and identify high-quality prompts, which is costly, inefficient, and subjective. Meanwhile, existing continuous prompt optimization methods improve the performance by learning the ideal prompts through the gradient information of PLMs, whose high computational cost, and low readability and generalizability are often concerning. To address the research gap, we propose a Dialogue-comprised Policy-gradient-based Discrete Prompt Optimization ($DP_2O$) method. We first design a multi-round dialogue alignment strategy for readability prompt set generation based on GPT-4. Furthermore, we propose an efficient prompt screening metric to identify high-quality prompts with linear complexity. Finally, we construct a reinforcement learning (RL) framework based on policy gradients to match the prompts to inputs optimally. By training a policy network with only 0.67% of the PLM parameter size on the tasks in the few-shot setting, $DP_2O$ outperforms the state-of-the-art (SOTA) method by 1.52% in accuracy on average on four open-source datasets. Moreover, subsequent experiments also demonstrate that $DP_2O$ has good universality, robustness, and generalization ability.

Figures (2)

• Figure 1: An illustration of the procedure and innovation Q&A of $\textsc{DP}_2\textsc{O}$. The procedure includes 1) Sampling the seed set from the training set via $\mathrm{SUE}$; 2) Constructing the prompt by multi-round dialogue with GPT-4 to align the inputs with the whole training set's distribution; 3) Employing an RL agent to match prompts with inputs to predict probabilistically; 4) Feeding all prompt-input pairs to a base PLM model for downstream tasks and ensemble predictions by probability weighting.
• Figure 2: Overview of $\textsc{DP}_2\textsc{O}$. In the prompt set construction stage, we use the multi-round dialogue alignment strategy to generate high-quality discrete prompts continuously. Given the seed inputs $\mathcal{Z}_{\textrm{seed}}$ with top-$m$$\mathrm{SUE}$ score, $\textsc{DP}_2\textsc{O}$ have a conversation with GPT-4, which has $round_{max}$ times outer loop and $m - 2$ times inner loop, to align inputs semantics with the training set. Then $\textsc{DP}_2\textsc{O}$ apply the assessment metric $\mathrm{SUE}$ to sort the prompts after dialogue to obtain the final candidate set $\mathcal{Z}_{\mathrm{cand}}$. We filter top-$h$ candidates as the final prompt set based on $\mathrm{SUE}$ score. In the prompt matching stage, we build a reinforcement learning framework to match the appropriate prompt from $\mathcal{X}$ for each input $z$ from $\mathcal{Z}_{\textrm{train}}$ with probability. The prompt-input pairs are fed into the base PLM to predict downstream tasks. The final prediction is the probability-weighted output of all pairs.