Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models
Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Fengyuan Liu, Marco Ciccone, Angelo Porrello, Simone Calderara
TL;DR
This paper tackles transferring task-specific updates across different pre-trained models by exploiting gradient-sign structure. The authors introduce GradFix, which masks the source task vector τ_A using the gradient signs g of the target model to produce δ^A = α (m ⊙ τ_A) and apply θ_B^{trans} = θ_B − δ^A, ensuring first-order descent. A theoretical guarantee ties the method to descent on the target loss, and a practical limited-data variant uses majority voting to estimate gradient signs from a small labeled set with a favorable concentration bound. Empirically, GradFix improves cross-model transfer in both vision and language tasks, often beating naive transfer and matching or exceeding few-shot fine-tuning, while requiring only a single gradient estimate pass to construct the mask. The approach promises substantial cost savings for adapting evolving foundation models, especially under strict data or compute constraints.
Abstract
When a new release of a foundation model is published, practitioners typically need to repeat full fine-tuning, even if the same task has already been solved in the previous version. A promising alternative is to reuse the parameter changes (i.e., task vectors) that capture how a model adapts to a specific task. However, they often fail to transfer across different pre-trained models due to their misaligned parameter space. In this work, we show that the key to successful transfer lies in the sign structure of the gradients of the new model. Based on this insight, we propose GradFix, a novel method that approximates the ideal gradient sign structure and leverages it to transfer knowledge using only a handful of labeled samples. Notably, this requires no additional fine-tuning: the adaptation is achieved by computing a few gradients at the target model and masking the source task vector accordingly. This yields an update that is locally aligned with the target loss landscape, effectively rebasing the task vector onto the new pre-training. We provide a theoretical guarantee that our method ensures first-order descent. Empirically, we demonstrate significant performance gains on vision and language benchmarks, consistently outperforming naive task vector addition and few-shot fine-tuning.