Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment
Kazuhiko Kawamoto, Atsuhiro Endo, Hiroshi Kera
TL;DR
The paper tackles the problem of transferring task vectors between models that are pre-trained on different data distributions, where direct transfer fails due to misaligned parameter spaces. It introduces a few-shot orthogonal similarity transformation that aligns source task updates into the target parameter space while preserving the updates' norm and rank, enabling modular, reusable task vectors. Task vectors are constructed via selective fine-tuning (embedding or LoRA) on a source Vision Transformer, then mapped to the target coordinates using layer-wise orthogonal matrices U_l. Empirical results on ViT-B/32 models pretrained on YFCC100M and LAION400M across eight datasets show substantial transfer gains, approaching the performance of full few-shot fine-tuning while preserving task-vector modularity. The approach offers a practical pathway to cross-domain task transfer with minimal supervision and invites future work on zero-shot alignment.
Abstract
Task arithmetic enables efficient model editing by representing task-specific changes as vectors in parameter space. Task arithmetic typically assumes that the source and target models are initialized from the same pre-trained parameters. This assumption limits its applicability in cross-model transfer settings, where models are independently pre-trained on different datasets. To address this challenge, we propose a method based on few-shot orthogonal alignment, which aligns task vectors to the parameter space of a differently pre-trained target model. These transformations preserve key properties of task vectors, such as norm and rank, and are learned using only a small number of labeled examples. We evaluate the method using two Vision Transformers pre-trained on YFCC100M and LAION400M, and test on eight classification datasets. Experimental results show that our method improves transfer accuracy over direct task vector application and achieves performance comparable to few-shot fine-tuning, while maintaining the modularity and reusability of task vectors. Our code is available at https://github.com/kawakera-lab/CrossModelTransfer.