IDPG: An Instance-Dependent Prompt Generation Method

TL;DR

This work addresses the inefficiency of fine-tuning large pre-trained LMs by proposing Instance-Dependent Prompt Generation (IDPG), which generates per-instance prompts conditioned on the input and task. IDPG uses a lightweight generator G to produce prompts from a sentence representation M(x) and a two-layer bottleneck, further compressed with Parameterized Hypercomplex Multiplication (PHM) layers, enabling only ~134K trainable parameters per task. Across ten NLU tasks, IDPG consistently outperforms fixed-prompt prompt-tuning baselines and achieves performance on par with or close to adapter-based methods like Compacter, while using far fewer trainable parameters. Extensive ablations show that PHM-based generators, even with lightweight sentence encoders like GloVe, provide robust gains, and multi-layer prompt insertion further enhances performance and scalability, especially in low-resource settings. The results establish instance-specific prompts as a practical, efficient alternative for transferring knowledge from large LMs to diverse downstream tasks.

Abstract

Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. It freezes the pre-trained language model and only optimizes a few task-specific prompts. In this paper, we propose a conditional prompt generation method to generate prompts for each input instance, referred to as the Instance-Dependent Prompt Generation (IDPG). Unlike traditional prompt tuning methods that use a fixed prompt, IDPG introduces a lightweight and trainable component to generate prompts based on each input sentence. Extensive experiments on ten natural language understanding (NLU) tasks show that the proposed strategy consistently outperforms various prompt tuning baselines and is on par with other efficient transfer learning methods such as Compacter while tuning far fewer model parameters.

Figures (7)

• Figure 1: Comparison between three different multi-layer generator models (S, M, L versions), and comparison between taking layer 0's output or previous layer's output as input.
• Figure 2: The number of pairs of each group in Top-200 cosine similarity ranking. More results can be found in Appendix \ref{['appendix-cos']}.
• Figure 3: Length difference of GLUE sentence pair datasets.
• Figure 4: Insertion positions for sentence-pair tasks.
• Figure 5: Impact of prompt position on (a) downstream tasks; (b) supplementary training phase.