Revisiting Parameter-Efficient Tuning: Are We Really There Yet?

TL;DR

This paper re-evaluates parameter-efficient tuning (PETuning) methods for pretrained language models under a fair evaluation protocol. It demonstrates that previous conclusions were inflated by data leakage from dev/test reuse and by instability across seeds; when properly controlled, finetuning often outperforms PETuning in medium- and high-resource settings. The authors analyze stability factors and show that reducing trainable parameter counts and increasing training iterations improve reliability, while prefix-tuning is particularly unstable. The work provides practical guidelines for fair benchmarking and suggests that PETuning is best suited for low-resource tasks, with a need for further advances to close the gap with full finetuning.

Abstract

Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs). By tuning just a fraction amount of parameters comparing to full model finetuning, PETuning methods claim to have achieved performance on par with or even better than finetuning. In this work, we take a step back and re-examine these PETuning methods by conducting the first comprehensive investigation into the training and evaluation of them. We found the problematic validation and testing practice in current studies, when accompanied by the instability nature of PETuning methods, has led to unreliable conclusions. When being compared under a truly fair evaluation protocol, PETuning cannot yield consistently competitive performance while finetuning remains to be the best-performing method in medium- and high-resource settings. We delve deeper into the cause of the instability and observed that the number of trainable parameters and training iterations are two main factors: reducing trainable parameters and prolonging training iterations may lead to higher stability in PETuning methods.

Figures (13)

• Figure 1: The relative performance difference of PETuning methods, i.e., Adapter, prefix tuning (PT), LoRA, BitFit, comparing with the full finetuning (FT) over different training data size of 12 tasks from GLUE and SuperGLUE. The tasks and their split into the three resource bands are illustrated in \ref{['sec:tasksetup']}. The size of each point denotes the standard deviation and the colours of PETuning methods denote the percentage of trainable parameters over different tasks compared to full finetuning. The key takeaway message is that PETuning methods outperform finetuning only in the low-resource tasks but remain on par or behind in medium and high-resource settings.
• Figure 2: Comparing early stopped points selected by RTE$_{1-2}$ and RTE$_{2-2}$, i.e., checkpoints with the best accuracy scores from dev.1 and dev.2 over training epochs. The markers denote the epochs selected by early stopping. Comparing the two checkpoint results on dev.2 (i.e. test performance), the RTE$_{2-2}$ (same set for test and dev) checkpoint usually shows higher performance than the checkpoint selected in RTE$_{1-2}$ by a large gap.
• Figure 3: Relative performance differences of prefix tuning (PT) over full finetuning (FT) on the upper bounds of multi-run results. PT achieves close upper bounds compared with FT on most of the 12 tasks.
• Figure 4: The experimental results over 20 different random seeds across CB, COPA, WSC, and RTE datasets, where finetuning and PETuning methods show large instability. The dashed rhombuses denote the mean (horizontal dashed line) and standard deviation (vertical distance).
• Figure 5: Performance probability density curves of Adapter, prefix tuning (PT), and LoRA over small, medium, and large parameter scales on COPA task across 20 runs. (See the numerical results and analyses in \ref{['sec:parameter_ana']}.)