Parameter-Efficient Fine-Tuning With Adapters

TL;DR

This work addresses the high resource cost of domain and task adaptation in language models by exploring adapter-based fine-tuning within the UniPELT framework, augmented by a PromptTuning Layer. Across GLUE, four-domain domain data, and SQuAD, the proposed methods achieve competitive performance while dramatically reducing the number of trainable parameters compared to full fine-tuning and prior DAPT/DAPT+TAPT approaches. The results show that parameter-efficient adapters are especially effective in domains with low vocabulary overlap with the base model, and that layer-wise stacking of UniPELT adapters often yields the best trade-off between performance and efficiency. The study also reveals task-specific nuances, such as SQuAD behavior, suggesting directions for future optimization of adapter architectures and configurations for generation-oriented tasks.

Abstract

In the arena of language model fine-tuning, the traditional approaches, such as Domain-Adaptive Pretraining (DAPT) and Task-Adaptive Pretraining (TAPT), although effective, but computational intensive. This research introduces a novel adaptation method utilizing the UniPELT framework as a base and added a PromptTuning Layer, which significantly reduces the number of trainable parameters while maintaining competitive performance across various benchmarks. Our method employs adapters, which enable efficient transfer of pretrained models to new tasks with minimal retraining of the base model parameters. We evaluate our approach using three diverse datasets: the GLUE benchmark, a domain-specific dataset comprising four distinct areas, and the Stanford Question Answering Dataset 1.1 (SQuAD). Our results demonstrate that our customized adapter-based method achieves performance comparable to full model fine-tuning, DAPT+TAPT and UniPELT strategies while requiring fewer or equivalent amount of parameters. This parameter efficiency not only alleviates the computational burden but also expedites the adaptation process. The study underlines the potential of adapters in achieving high performance with significantly reduced resource consumption, suggesting a promising direction for future research in parameter-efficient fine-tuning.