Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection
Shiyun Xu, Zhiqi Bu
TL;DR
The paper addresses how to adaptively select the active parameter groups for parameter-efficient fine-tuning (PEFT) to balance model performance and trainable parameter budgets. It introduces AdaPEFT, a Hessian-informed framework that formulates subset selection as a multi-task Pareto optimization and reduces it to a $0$-$1$ knapsack problem via an $oldsymbol{\e}$-constraint, gradient-based inner minimization, and Taylor approximations. By deriving Hessian-informed loss reductions and proposing efficient estimation methods (quadratic curve fitting, lazy updates) as well as both exact and approximate knapsack solvers, the approach identifies influential parameter groups whose utility transfers across training horizons and model sizes. Empirical analyses show that influential groups vary substantially across models and tasks yet exhibit cross-model consistency, enabling AdaPEFT to transfer from small to large models with limited budget and to approximate the Pareto frontier closely compared to fixed-PEFT baselines. Overall, AdaPEFT offers a scalable, principled path to adaptive PEFT with strong practical impact for efficiently adapting large pre-trained models across diverse domains.
Abstract
Parameter-efficient fine-tuning (PEFT) is a highly effective approach for adapting large pre-trained models to downstream tasks with minimal computational overhead. At the core, PEFT methods freeze most parameters and only trains a small subset (say $<0.1\%$ of total parameters). Notably, different PEFT methods select different subsets, resulting in varying levels of performance. This variation prompts a key question: how to effectively select the most influential subset to train? We formulate the subset selection as a multi-task problem: maximizing the performance and minimizing the number of trainable parameters. We leverage a series of transformations -- including $ε$-constraint method and second-order Taylor approximation -- to arrive at the classical 0-1 knapsack problem, which we solve through the lens of Pareto optimality. Consequently, we propose AdaPEFT, a Hessian-informed PEFT that adapts to various tasks and models, in which the selected subset empirically transfers across training horizons and model sizes.