Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Merging with Functional Dual Anchors

Kexuan Shi, Yandong Wen, Weiyang Liu

TL;DR

This work introduces Functional Dual Anchors (FDAs), a novel input-space framework for model merging that shifts task knowledge encoding from parameters to synthetic inputs whose gradients align with task vectors. FDAs are constructed by gradient-matching objectives in the input space and then used to optimize the merged model, either independently or in combination with traditional parameter-space merges (e.g., TA, TSV, WUDI). The approach includes principled initialization, a layer-wise practical implementation, and extensive experiments across vision, language, and auto-regressive models, demonstrating notable gains in multi-task performance and GLUE scores. Analyses show FDAs develop a long-tailed energy distribution and align with real-data subspaces, while ablations guide choices on initialization, shape, distance metrics, and construction steps. Overall, FDAs offer a robust, flexible, and complementary path to consolidate cross-task knowledge without retraining on task data, with practical benefits for large foundation models.

Abstract

Model merging is an efficient post-training strategy for integrating knowledge from multiple finetuned checkpoints of a shared foundation model. Existing methods operate in the parameter space, combining task vectors to mitigate conflicts, but remain constrained by parameter inconsistencies. We propose Functional Dual Anchors (FDAs), a framework that instead models the input-representation space. FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model. This perspective bridges joint multi-task training and post-hoc merging, offering both robustness and flexibility. We further introduce a principled initialization scheme and show that FDAs are complementary to parameter-space model merging. Comprehensive experiments demonstrate the effectiveness of FDAs in model merging.

Model Merging with Functional Dual Anchors

TL;DR

This work introduces Functional Dual Anchors (FDAs), a novel input-space framework for model merging that shifts task knowledge encoding from parameters to synthetic inputs whose gradients align with task vectors. FDAs are constructed by gradient-matching objectives in the input space and then used to optimize the merged model, either independently or in combination with traditional parameter-space merges (e.g., TA, TSV, WUDI). The approach includes principled initialization, a layer-wise practical implementation, and extensive experiments across vision, language, and auto-regressive models, demonstrating notable gains in multi-task performance and GLUE scores. Analyses show FDAs develop a long-tailed energy distribution and align with real-data subspaces, while ablations guide choices on initialization, shape, distance metrics, and construction steps. Overall, FDAs offer a robust, flexible, and complementary path to consolidate cross-task knowledge without retraining on task data, with practical benefits for large foundation models.

Abstract

Model merging is an efficient post-training strategy for integrating knowledge from multiple finetuned checkpoints of a shared foundation model. Existing methods operate in the parameter space, combining task vectors to mitigate conflicts, but remain constrained by parameter inconsistencies. We propose Functional Dual Anchors (FDAs), a framework that instead models the input-representation space. FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model. This perspective bridges joint multi-task training and post-hoc merging, offering both robustness and flexibility. We further introduce a principled initialization scheme and show that FDAs are complementary to parameter-space model merging. Comprehensive experiments demonstrate the effectiveness of FDAs in model merging.
Paper Structure (27 sections, 1 theorem, 29 equations, 15 figures, 8 tables, 1 algorithm)

This paper contains 27 sections, 1 theorem, 29 equations, 15 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. A Model Merging Framework with Functional Dual Anchors
  3. Preliminaries and Background
  4. FDA Construction: Knowledge Projection via Gradient Matching
  5. Parameter Optimization: Leveraging FDAs for Multi-task Model Merging
  6. Practical Implementation
  7. Towards Understanding FDA-encoded Knowledge
  8. Main Experiments and Results
  9. Experimental Settings
  10. Experiments on Vision and Language Models
  11. Ablation and Explorative Study
  12. Comparison of Initialization Schemes
  13. The Shape of FDAs
  14. The Effect of Distance functions
  15. Optimization Steps in FDA Construction
  16. ...and 12 more sections

Key Result

Proposition 2.1

Under the above setting, for any iteration $t$, ${\bm{x}}_t$ can be expressed as the linear combination of $\{{\bm{u}}_i\}_{i=1}^d$. Specifically, the coefficient $c_t^i$ associated with basis vector ${\bm{u}}_i$ is given by $c_t^i \;=\; c_0^i \prod_{j=1}^t \bigl(1 - \gamma_j \lambda_i \bigr),$ wher

Figures (15)

  • Figure 1: Illustration of our input-space model merging framework using FDAs. On the left, we compare multi-task joint training, task arithmetic and FDA. Inspired by joint training, FDA models the knowledge in the input space. $\theta_A=FT(\bm{X}_A,\bm{\theta}_0)$ denotes the model finetuned by the task data $\bm{X}_A$ from the initial model $\bm{\theta}_0$ with some loss function.
  • Figure 2: Comparison between task arithmetic and FDAs on the loss landscape of the pretrained across all 8 downstream datasets. FDAs can effectively follow the loss landscape and guide the model toward better local minima.
  • Figure 3: Evolution of Normalized singular values of FDAs in the FDA construction. We visualize the results of FDAs from the $12$-th layer of the ViT-B/32 checkpoint on MNIST. $\sigma=10^1$ denotes FDAs initialized by sampling from $\mathcal{N}(\mathbf{0}, {\bm{I}}_d)$ and scaling by $10^1$; “Weight” denotes FDAs initialized from linear weight. FDAs of different initialization schemes tend to evolve into long-tailed structures.
  • Figure 4: Average loss curves.
  • Figure 5: Evolution of subspace similarity of FDAs in the FDA construction. We visualize the results of FDAs from the $12$-th layer of the ViT-B/32 checkpoint. $\sigma=10^1$ denotes the FDAs initialized by sampling from $\mathcal{N}(\mathbf{0}, {\bm{I}}_d)$ and scaling by $10^1$; “Weight” denotes the FDAs initialized from linear weight. FDAs of different initialization schemes tend to align the subspace spanned by real data.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Proposition 2.1
  • proof
  • proof : Proof of Proposition \ref{['prop:spectral']}