Continuous Prompt Generation from Linear Combination of Discrete Prompt Embeddings

TL;DR

The paper addresses the interpretability challenges of continuous prompts in large language models by proposing a method that constructs a continuous prompt as a learned linear combination of a small set of discrete prompt embeddings. A feed-forward network predicts the combination weights, enabling a continuous prompt that inherits the interpretability of discrete prompts while retaining optimization advantages. Evaluations on the ARC Challenge using a BART backbone show cross-entropy loss reductions and weight patterns that align with semantically meaningful prompts, suggesting improved interpretability without sacrificing performance. This approach offers a path toward safer, more transparent prompt engineering and highlights the importance of basis selection and task coverage for generalization.

Abstract

The wayward quality of continuous prompts stresses the importance of their interpretability as unexpected and unpredictable behaviors appear following training, especially in the context of large language models automating people-sensitive tasks such as resume screening. In this paper we present a novel method of constructing continuous prompts via discrete prompt embeddings and evaluate improvements to continuous prompt interpretability and inference accuracy. For a set of manually designed discrete prompts $\mathcal{D}$, which we tokenize and embed each into tensor form, we train a model to predict the weights such that the linear combinations of those prompts correspond to higher performance on natural language understanding tasks.

Figures (3)

• Figure 1: Flow diagram of our Proposed Approach
• Figure 2: Performance of BART pre-trained model on 20 epochs batch size 2
• Figure 3: Performance of BART pre-trained model on 20 epochs batch size 10