Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transporting Task Vectors across Different Architectures without Training

Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Angelo Porrello, Simone Calderara

TL;DR

Theseus tackles the problem of transferring task-specific updates across models with differing widths and architectures without additional training. It treats task updates by their functional effect on activations, aligning source and target representations with orthogonal Procrustes maps and deriving a closed-form transport rule $\tau_B = T_{\text{out}} \tau_A T_{\text{in}}^{\top}$. The approach is validated across vision and language tasks, demonstrating training-free transfers that outperform naive baselines and remain competitive with gradient-based methods in identical-architecture settings, while also serving as a warm-start for faster fine-tuning. The work advocates a functional notion of task identity, enabling efficient modular reuse of task knowledge across evolving model families and potentially reducing computational costs in deploying large pre-trained systems.

Abstract

Adapting large pre-trained models to downstream tasks often produces task-specific parameter updates that are expensive to relearn for every model variant. While recent work has shown that such updates can be transferred between models with identical architectures, transferring them across models of different widths remains largely unexplored. In this work, we introduce Theseus, a training-free method for transporting task-specific updates across heterogeneous models. Rather than matching parameters directly, we characterize a task update by the functional effect it induces on intermediate representations. We formalize task-vector transport as a functional matching problem on observed activations and show that, after aligning representation spaces via orthogonal Procrustes analysis, it admits a stable closed-form solution that preserves the geometry of the update. We evaluate Theseus on vision and language models across different widths, showing consistent improvements over strong baselines without additional training or backpropagation. Our results show that task updates can be meaningfully transferred across architectures when task identity is defined functionally rather than parametrically.

Transporting Task Vectors across Different Architectures without Training

TL;DR

Theseus tackles the problem of transferring task-specific updates across models with differing widths and architectures without additional training. It treats task updates by their functional effect on activations, aligning source and target representations with orthogonal Procrustes maps and deriving a closed-form transport rule . The approach is validated across vision and language tasks, demonstrating training-free transfers that outperform naive baselines and remain competitive with gradient-based methods in identical-architecture settings, while also serving as a warm-start for faster fine-tuning. The work advocates a functional notion of task identity, enabling efficient modular reuse of task knowledge across evolving model families and potentially reducing computational costs in deploying large pre-trained systems.

Abstract

Adapting large pre-trained models to downstream tasks often produces task-specific parameter updates that are expensive to relearn for every model variant. While recent work has shown that such updates can be transferred between models with identical architectures, transferring them across models of different widths remains largely unexplored. In this work, we introduce Theseus, a training-free method for transporting task-specific updates across heterogeneous models. Rather than matching parameters directly, we characterize a task update by the functional effect it induces on intermediate representations. We formalize task-vector transport as a functional matching problem on observed activations and show that, after aligning representation spaces via orthogonal Procrustes analysis, it admits a stable closed-form solution that preserves the geometry of the update. We evaluate Theseus on vision and language models across different widths, showing consistent improvements over strong baselines without additional training or backpropagation. Our results show that task updates can be meaningfully transferred across architectures when task identity is defined functionally rather than parametrically.
Paper Structure (22 sections, 1 theorem, 22 equations, 2 figures, 9 tables, 1 algorithm)

This paper contains 22 sections, 1 theorem, 22 equations, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Properties and Scope
  5. Experiments
  6. Experimental Setting
  7. Transfer across Mismatched Widths
  8. Transfer on Identical Architectures
  9. Transfer across Depth and Width
  10. Theseus as Warm-Start for Fine-Tuning
  11. Conclusions
  12. Closed-form Derivation of the Transport Rule
  13. Injectivity and Rank Conditions
  14. Relation to Pseudo-Inverse Solutions
  15. Experimental setting
  16. ...and 7 more sections

Key Result

Lemma 2.1

Let $H_{\mathrm{in},A} \in \mathbb{R}^{M \times d_{\mathrm{in},A}}$ and $H_{\mathrm{out},A} \in \mathbb{R}^{M \times d_{\mathrm{out},A}}$ have full column rank. Then the linear map is injective.

Figures (2)

  • Figure 1: Task-vector transport via orthogonal alignment. Procrustes rotations identify shared structure between representation spaces, enabling training-free transfer of task updates across models with mismatched dimensions.
  • Figure 2: Comparative Analysis of Convergence Trends. Global average validation loss and accuracy across eight datasets. We compare standard fine-tuning from baseline $\theta_B$ against initialization via our approach.

Theorems & Definitions (2)

  • Lemma 2.1
  • proof