Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning

TL;DR

This work tackles the limitations of parameter-efficient fine-tuning (PEFT), especially few-shot adaptation and cross-domain generalization. It introduces a general priming framework that places an upstream priming stage (trained with multitask learning or meta-learning) between pre-training and downstream PEFT, enabling faster and more robust adaptation to new tasks. Experiments on the CrossFit Challenge with 160 tasks show that priming generally boosts performance and accelerates convergence, with Priming PLM–only strategies often achieving top results. Overall, the study demonstrates a practical, resource-efficient pathway to improve cross-domain few-shot learning for PLMs by aligning pre-training with downstream PEFT through priming.

Abstract

Parameter-efficient (PE) methods (like Prompts or Adapters) for adapting pre-trained language models (PLM) to downstream tasks have been popular recently. However, hindrances still prevent these methods from reaching their full potential. For example, two significant challenges are few-shot adaptation and cross-task generalization. To tackle these issues, we propose a general PE priming framework to enhance and explore the few-shot adaptation and generalization ability of PE methods. In this framework, PLMs are primed with PE methods for rapidly adapting to various target tasks. To evaluate the generalization ability of these PE methods, we conduct experiments on a few-shot cross-domain benchmark containing 160 diverse NLP tasks. Our experiment not only reveals the best priming strategy but also verifies that priming facilitates the adaptation to target tasks.

Figures (4)

• Figure 1: We propose a general framework to improve the performance of parameter-efficient fine-tuning. We prime the PLM with source tasks for parameter-efficient methods.
• Figure 2: Illustration of Alg. \ref{['alg:cap']}
• Figure 3: This figure shows the parameter efficiency of different baselines.
• Figure 4: The loss curves of w & w/o priming prompt in the fine-tuning stage.