Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation

TL;DR

The paper tackles controllable text generation for large language models, aiming to be data- and parameter-efficient on smaller models. It proposes Plug and Play with Prompts (PPP), which attaches learnable soft-prompt embeddings to the input and optimizes them via a discriminator-based objective plus a fluency anchor, without altering the generator. PPP achieves strong style control on SST-5, Yelp, GYAFC, and JIGSAW with only a few hundred training examples and generalizes to out-of-domain data, including detoxifying toxic outputs. Compared to zero-shot, few-shot, PPLM, and GeDi baselines, PPP delivers superior or competitive style control with preserved fluency and high data efficiency, and it remains memory-efficient by only learning a small set of prompt parameters.

Abstract

Transformer-based Large Language Models (LLMs) have shown exceptional language generation capabilities in response to text-based prompts. However, controlling the direction of generation via textual prompts has been challenging, especially with smaller models. In this work, we explore the use of Prompt Tuning to achieve controlled language generation. Generated text is steered using prompt embeddings, which are trained using a small language model, used as a discriminator. Moreover, we demonstrate that these prompt embeddings can be trained with a very small dataset, with as low as a few hundred training examples. Our method thus offers a data and parameter efficient solution towards controlling language model outputs. We carry out extensive evaluation on four datasets: SST-5 and Yelp (sentiment analysis), GYAFC (formality) and JIGSAW (toxic language). Finally, we demonstrate the efficacy of our method towards mitigating harmful, toxic, and biased text generated by language models.

Figures (2)

• Figure 1: Model architecture of the proposed PPP method. (a) To control the direction, the embeddings are generated autoregressively using the prompt and source embeddings. The generated embeddings are concatenated with the source embeddings and fed as input to the discriminator, which generates the classification loss. (b) To maintain fluency, the source embeddings (without the trained prompt embeddings) are fed to the generator and the output embeddings are autoregressively generated. The cross entropy is calculated between the prompted and non-prompted output probabilities at each generation step and averaged over all steps to get the fluency loss. This loss acts as an "anchor", preventing the generated text from deviating too far from the original text and becoming nonsensical.
• Figure 2: Plots showing performance (style control) as a function of (a) Prompt Length, and (b) Model Size (parameters).