Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models

Aofei Chang, Jiaqi Wang, Han Liu, Parminder Bhatia, Cao Xiao, Ting Wang, Fenglong Ma

TL;DR

BIPEFT addresses the cost and inefficiency of fine-tuning large language models by automatically designing PEFT configurations under a parameter budget. It introduces a budget-guided iterative search that disentangles binary module existence from rank-dimension searches, with budget-aware early stopping and adaptive selection for both modules and dimensions. The method relies on a differential NAS framework using two interconnected architecture spaces and a Gumbel-Softmax-based optimization, achieving strong performance on GLUE and SuperGLUE with substantially reduced search costs. BIPEFT demonstrates superior efficiency and generalization, offering a practical solution for scalable PEFT in real-world downstream tasks.

Abstract

Parameter Efficient Fine-Tuning (PEFT) offers an efficient solution for fine-tuning large pretrained language models for downstream tasks. However, most PEFT strategies are manually designed, often resulting in suboptimal performance. Recent automatic PEFT approaches aim to address this but face challenges such as search space entanglement, inefficiency, and lack of integration between parameter budgets and search processes. To overcome these issues, we introduce a novel Budget-guided Iterative search strategy for automatic PEFT (BIPEFT), significantly enhancing search efficiency. BIPEFT employs a new iterative search strategy to disentangle the binary module and rank dimension search spaces. Additionally, we design early selection strategies based on parameter budgets, accelerating the learning process by gradually removing unimportant modules and fixing rank dimensions. Extensive experiments on public benchmarks demonstrate the superior performance of BIPEFT in achieving efficient and effective PEFT for downstream tasks with a low parameter budget.

BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models

TL;DR

BIPEFT addresses the cost and inefficiency of fine-tuning large language models by automatically designing PEFT configurations under a parameter budget. It introduces a budget-guided iterative search that disentangles binary module existence from rank-dimension searches, with budget-aware early stopping and adaptive selection for both modules and dimensions. The method relies on a differential NAS framework using two interconnected architecture spaces and a Gumbel-Softmax-based optimization, achieving strong performance on GLUE and SuperGLUE with substantially reduced search costs. BIPEFT demonstrates superior efficiency and generalization, offering a practical solution for scalable PEFT in real-world downstream tasks.

Abstract

Parameter Efficient Fine-Tuning (PEFT) offers an efficient solution for fine-tuning large pretrained language models for downstream tasks. However, most PEFT strategies are manually designed, often resulting in suboptimal performance. Recent automatic PEFT approaches aim to address this but face challenges such as search space entanglement, inefficiency, and lack of integration between parameter budgets and search processes. To overcome these issues, we introduce a novel Budget-guided Iterative search strategy for automatic PEFT (BIPEFT), significantly enhancing search efficiency. BIPEFT employs a new iterative search strategy to disentangle the binary module and rank dimension search spaces. Additionally, we design early selection strategies based on parameter budgets, accelerating the learning process by gradually removing unimportant modules and fixing rank dimensions. Extensive experiments on public benchmarks demonstrate the superior performance of BIPEFT in achieving efficient and effective PEFT for downstream tasks with a low parameter budget.

Paper Structure

This paper contains 28 sections, 9 equations, 3 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview
  5. Budget-aware Trigger Generation
  6. Module Sensitivity Estimation
  7. Trigger Generation
  8. Binary Module Selection
  9. Expected Parameters Estimation
  10. Adaptive Module Selection
  11. Multiple Rank Dimension Selection
  12. Dimension Stability Estimation
  13. Adaptive Dimension Selection
  14. Iterative Optimization
  15. Experiments
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Search space comparison among different automatic PEFT approaches when searching on low-rank adaption (LoRA) hu2022lora. S$^3$Delta hu2022sparse uses a $\{0,1\}$ binary space to determine whether the module is kept. AutoPEFT zhou-etal-2024-autopeft uses a multi-dimensional space to search module existence and dimension ranks simultaneously, where 0 means the module will be removed. The proposed BIPEFT uses a novel iterative search strategy to disentangle the binary module search and the rank dimension search.
  • Figure 2: Overview of the proposed BIPEFT, which conducts an iterative search on disentanglement search spaces with novel module and rank dimension selection strategies to accelerate search efficiency.
  • Figure 3: Performance vs. different levels of budget.