Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts

TL;DR

The paper investigates why reparameterization in prefix-tuning yields strong performance by revealing a hidden shared structure between prefix keys and values, rooted in a Mixture of Experts interpretation of attention. It provides theoretical convergence results showing that shared structures substantially improve sample efficiency over nonshared prompts, in both simple linear and one-layer neural network settings, with rates such as $O_P(\,\sqrt{\log(n)/n}\,) $ for singleton cells and $O_P((\log n / n)^{1/4})$ for multi-cell cells. Empirically, prefix-tuning with reparameterization achieves competitive results against full fine-tuning across visual and language tasks, and analyses on FGVC/VTAB-1K demonstrate meaningful gains from shared structures, while prompt-tuning exhibits analogous benefits. These findings offer both theoretical justification and practical guidance for designing more efficient prompt-based fine-tuning methods, with implications for MoE-based architectures and beyond.

Abstract

Prompt-based techniques, such as prompt-tuning and prefix-tuning, have gained prominence for their efficiency in fine-tuning large pre-trained models. Despite their widespread adoption, the theoretical foundations of these methods remain limited. For instance, in prefix-tuning, we observe that a key factor in achieving performance parity with full fine-tuning lies in the reparameterization strategy. However, the theoretical principles underpinning the effectiveness of this approach have yet to be thoroughly examined. Our study demonstrates that reparameterization is not merely an engineering trick but is grounded in deep theoretical foundations. Specifically, we show that the reparameterization strategy implicitly encodes a shared structure between prefix key and value vectors. Building on recent insights into the connection between prefix-tuning and mixture of experts models, we further illustrate that this shared structure significantly improves sample efficiency in parameter estimation compared to non-shared alternatives. The effectiveness of prefix-tuning across diverse tasks is empirically confirmed to be enhanced by the shared structure, through extensive experiments in both visual and language domains. Additionally, we uncover similar structural benefits in prompt-tuning, offering new perspectives on its success. Our findings provide theoretical and empirical contributions, advancing the understanding of prompt-based methods and their underlying mechanisms.

Figures (3)

• Figure 1: Reparameterization defines both the prefix key ${\bm p}^K_i$ and value ${\bm p}^V_i$ as functions of shared parameters ${\bm p}'_i$, transformed by $g_\theta$. This introduces parameter sharing between the score functions and expert parameters in the MoE framework in attention. The gating function computes expert weights based on score functions, and the MoE output is a weighted average of all expert outputs.
• Figure 2: Comparison of prefix-tuning across three configurations: Deep-share, Simple-share, and No-share, referred to as Deep, Simple, and No, respectively, on FGVC benchmarks.
• Figure 3: GradCAM visualization of Deep-share and No-share on FGVC tasks. Red regions indicate high-class activation scores. From left to right: input image after standard data augmentation, GradCAM output for No-share, and GradCAM output for Deep-share.

Theorems & Definitions (6)

• Theorem 4.1
• Theorem 4.2
• Theorem 4.3
• Proposition B.1
• Proposition B.2
• Lemma C.1