LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models

TL;DR

This paper tackles efficient adaptation of large language models by reducing the trainable footprint of soft prompts. It introduces Low-rank Prompt Tuning (LoPT), which enforces a low-rank structure on the soft-prompt matrix $X \in \mathbb{R}^{n \times d}$ and presents two implementations: LoPT-1 ($X=UV$, with $U \in \mathbb{R}^{n \times r}$, $V \in \mathbb{R}^{r \times d}$) and LoPT-2 ($X=\sigma(X_0 U) V$, with $X_0 \in \mathbb{R}^{n \times d}$, $U \in \mathbb{R}^{d \times r}$, $V \in \mathbb{R}^{r \times d}$). The approach achieves substantial parameter reductions (e.g., from $nd$ to $r(n+d)$ or $2rd$) while maintaining competitive accuracy across SST-2, AGNews, and SuperGLUE tasks on GPT-2 large and T5-base models. Ablation shows that $n$ and $r$ interact to determine performance, with best results near $n=20$ and $r$ around 2–5; limitations include reliance on the low-rank assumption. Overall, LoPT offers a practical, scalable PEFT technique for prompt tuning in large LMs.

Abstract

In prompt tuning, a prefix or suffix text is added to the prompt, and the embeddings (soft prompts) or token indices (hard prompts) of the prefix/suffix are optimized to gain more control over language models for specific tasks. This approach eliminates the need for hand-crafted prompt engineering or explicit model fine-tuning. Prompt tuning is significantly more parameter-efficient than model fine-tuning, as it involves optimizing partial inputs of language models to produce desired outputs. In this work, we aim to further reduce the amount of trainable parameters required for a language model to perform well on specific tasks. We propose Low-rank Prompt Tuning (LoPT), a low-rank model for prompts that achieves efficient prompt optimization. The proposed method demonstrates similar outcomes to full parameter prompt tuning while reducing the number of trainable parameters by a factor of 5. It also provides promising results compared to the state-of-the-art methods that would require 10 to 20 times more parameters.