Exploring Universal Intrinsic Task Subspace via Prompt Tuning

TL;DR

The paper investigates why pre-trained language models can readily adapt to diverse NLP tasks with limited data. It introduces Intrinsic Prompt Tuning (IPT), a two-stage pipeline (MSF and IST) that discovers a shared, low-dimensional intrinsic task subspace for task adaptations and then tunes only a small set of parameters within that subspace. Empirical results show that a 250-dimensional subspace learned from 100 tasks can recover most of the full prompt-tuning performance on both seen and unseen tasks, with improved stability and insights into task similarities. These findings offer a new lens on cross-task generalization and suggest practical avenues for efficient, stable prompt-based adaptation across many NLP tasks.

Abstract

Why can pre-trained language models (PLMs) learn universal representations and effectively adapt to broad NLP tasks differing a lot superficially? In this work, we empirically find evidence indicating that the adaptations of PLMs to various few-shot tasks can be reparameterized as optimizing only a few free parameters in a unified low-dimensional intrinsic task subspace, which may help us understand why PLMs could easily adapt to various NLP tasks with small-scale data. To find such a subspace and examine its universality, we propose an analysis pipeline called intrinsic prompt tuning (IPT). Specifically, we resort to the recent success of prompt tuning and decompose the soft prompts of multiple NLP tasks into the same low-dimensional nonlinear subspace, then we learn to adapt the PLM to unseen data or tasks by only tuning parameters in this subspace. In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace. Besides being an analysis tool, IPT could further help us improve the prompt tuning stability.

Figures (12)

• Figure 1: An illustration of a common low-dimensional intrinsic task subspace for diverse tasks. PLMs tune adaptive parameters to adapt to each task.
• Figure 2: Illustrations of (a) fine-tuning, (b) prompt tuning and (c,d) two components of IPT. We discriminate tunable parameters, fixed parameters and intermediate features with different colors.
• Figure 3: Relative performance of IPT at different dimension $d_{I}$ on three task splits (random, non-cls and cls). We report the relative performance of IPT compared with both prompt tuning ($E_\text{rel}^\text{PT}$) and fine-tuning ($E_\text{rel}^\text{FT}$).
• Figure 4: Comparisons ($E_\text{rel}^\text{FT}$) between IPT and other subspace-finding methods on $\mathcal{T}_{\mathrm{test}}$ of random task split.
• Figure 5: Impacts of the data scale.