Hard Prompts Made Interpretable: Sparse Entropy Regularization for Prompt Tuning with RL

TL;DR

This work addresses the interpretability gap in hard prompt tuning for black-box models by identifying detrimental Q-value estimation in RLPrompt and introducing Prompts made INterpretable (PIN) with sparse Tsallis entropy regularization. By filtering ignorable tokens and enforcing sparsity, PIN produces prompts that are both more natural and more task-relevant, while maintaining competitive performance across few-shot classification, unsupervised style transfer, and textual inversion. The approach reframes prompt learning as a constrained, sparse reinforcement learning problem, mitigating overfitting to improbable tokens and improving stability in Q-value estimation. Overall, PIN offers a principled, model-agnostic method for efficient, interpretable hard prompt discovery with potential broad impact on practical deployment of foundation models.

Abstract

With the advent of foundation models, prompt tuning has positioned itself as an important technique for directing model behaviors and eliciting desired responses. Prompt tuning regards selecting appropriate keywords included into the input, thereby adapting to the downstream task without adjusting or fine-tuning the model parameters. There is a wide range of work in prompt tuning, from approaches that directly harness the backpropagated gradient signals from the model, to those employing black-box optimization such as reinforcement learning (RL) methods. Our primary focus is on RLPrompt, which aims to find optimal prompt tokens leveraging soft Q-learning. While the results show promise, we have observed that the prompts frequently appear unnatural, which impedes their interpretability. We address this limitation by using sparse Tsallis entropy regularization, a principled approach to filtering out unlikely tokens from consideration. We extensively evaluate our approach across various tasks, including few-shot text classification, unsupervised text style transfer, and textual inversion from images. The results indicate a notable improvement over baselines, highlighting the efficacy of our approach in addressing the challenges of prompt tuning. Moreover, we show that the prompts discovered using our method are more natural and interpretable compared to those from other baselines.

Figures (10)

• Figure 1: Training curves on textual inversion from images task. For each dataset, training was conducted on 10 target images across 3 random seeds. The solid curves show the average CLIP Score across target images, with the shaded areas representing the standard error.
• Figure 2: Generated images from Stable Diffusion-v2 latent-diff, using the learned hard prompts (bottom) for the target image (left). We also showcase more qualitative examples in Appendix \ref{['app:qau']}.
• Figure 3: (a) Comparison with PIN-no-fluency at varying prompt length, and (b) Analysis on the effect of $k$.
• Figure 4: Training curve between PIN (ours) and RLPrompt. For each of the 5 few-shot training sets, 3 experiments were conducted. The graph depicts the average reward over the validation sets with standard error shading.
• Figure 5: Training curve between our PIN method and RLPrompt over various task models (e.g. OPT-125M / 350M / 1.3B). For each dataset, specifically for sentiment style transfer including negative-to-positive (0 to 1) and positive-to-negative (1 to 0), 4 experiments were conducted with different seeds. The graph depicts the average reward on the validation set with standard error shading.