Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies
Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, Fei Yang
TL;DR
This survey addresses the challenge of fine-tuning giant pre-trained models by systematizing Parameter-Efficient Fine-Tuning (PEFT) methods. It synthesizes over 100 works (2019–2024), frameworks, and cross-domain applications, organizing approaches into Additive, Reparameterized, Selective, Hybrid, Quantization, and Multi-task categories, with extensions for vision and diffusion models. The authors provide a theoretical framework, schematic visualizations, and comparative tables to clarify each method’s trainable parameter budget and applicability, including examples like LoRA, adapters, and prompt-tuning variants. A central contribution is the cross-domain perspective, showing how PEFT can reduce compute and storage while preserving or approximating full-finetuning performance, and offering concrete future directions such as multi-objective optimization, automated adapter design, continual learning, and privacy-aware tuning. The work highlights the practical significance of PEFT for sustainable AI research and real-world deployment of large multimodal models, emphasizing both methodological advances and implementation guidance across NLP, vision, and diffusion domains.
Abstract
The large models, as predicted by scaling raw forecasts, have made groundbreaking progress in many fields, particularly in natural language generation tasks, where they have approached or even surpassed human levels. However, the unprecedented scale of their parameters brings significant computational and storage costs. These large models require substantial computational resources and GPU memory to operate. When adapting large models to specific downstream tasks, their massive parameter scale poses a significant challenge in fine-tuning on hardware platforms with limited computational power and GPU memory. To address this issue, Parameter-Efficient Fine-Tuning (PEFT) offers a practical solution by efficiently adjusting the parameters of large pre-trained models to suit various downstream tasks. Specifically, PEFT adjusts the parameters of pre-trained large models to adapt to specific tasks or domains, minimizing the introduction of additional parameters and the computational resources required. This review mainly introduces the preliminary knowledge of PEFT, the core ideas and principles of various PEFT algorithms, the applications of PEFT, and potential future research directions. By reading this review, we believe that interested parties can quickly grasp the PEFT methodology, thereby accelerating its development and innovation.