mergetune: Continued fine-tuning of vision-language models

TL;DR

This work tackles catastrophic forgetting during fine-tuning of vision-language models by proposing MergeTune, a continued fine-tuning paradigm guided by linear mode connectivity to merge zero-shot and downstream solutions post hoc without architectural changes. It introduces a replay-free, second-order surrogate to enforce low-loss connectivity between the zero-shot and fine-tuned endpoints, enabling practical optimization on large pretraining data. Across base-to-novel, cross-dataset, domain generalisation, and robust fine-tuning tasks, MergeTune provides consistent performance gains and often surpasses ensemble baselines while maintaining lower inference costs. The approach is model-agnostic and applicable to prompts, adapters, or linear heads, offering a scalable way to recover pretrained knowledge while preserving downstream adaptation.

Abstract

Fine-tuning vision-language models (VLMs) such as CLIP often leads to catastrophic forgetting of pretrained knowledge. Prior work primarily aims to mitigate forgetting during adaptation; however, forgetting often remains inevitable during this process. We introduce a novel paradigm, \emph{continued fine-tuning (CFT)}, which seeks to recover pretrained knowledge after a zero-shot model has already been adapted. We propose a simple, model-agnostic CFT strategy (named MERGETUNE) guided by linear mode connectivity (LMC), which can be applied post hoc to existing fine-tuned models without requiring architectural changes. Given a fine-tuned model, we continue fine-tuning its trainable parameters (e.g., soft prompts or linear heads) to search for a continued model which has two low-loss paths to the zero-shot (e.g., CLIP) and the fine-tuned (e.g., CoOp) solutions. By exploiting the geometry of the loss landscape, the continued model implicitly merges the two solutions, restoring pretrained knowledge lost in the fine-tuned counterpart. A challenge is that the vanilla LMC constraint requires data replay from the pretraining task. We approximate this constraint for the zero-shot model via a second-order surrogate, eliminating the need for large-scale data replay. Experiments show that MERGETUNE improves the harmonic mean of CoOp by +5.6\% on base-novel generalisation without adding parameters. % We show \emph{the first time} superior performance than CLIP on both DTD and EuroSAT, on cross-dataset transfer. On robust fine-tuning evaluations, the LMC-merged model from MERGETUNE surpasses ensemble baselines with lower inference cost, achieving further gains and state-of-the-art results when ensembled with the zero-shot model. Our code is available at \href{https://github.com/Surrey-UP-Lab/MERGETUNE}{https://github.com/Surrey-UP-Lab/MERGETUNE}.

Figures (5)

• Figure 1: Cross-dataset generalisation shows no single PEFT method consistently outperforms CLIP across all 11 datasets, implying incomplete preservation of pretrained knowledge. Numbers in brackets (X/11) indicate X times a method underperforms CLIP.
• Figure 2: The proposed MergeTune(conceptual illustration). (Left) Before MergeTune Training: The zero-shot model $\hat{w}_{1}$ and fine-tuned $\hat{w}_{2}$ exist in separate minima with no low-loss connectivity. Linear interpolation between them (shown in the inset) reveals high barriers and induces a performance trade-off on base and novel classes. (Middle) During training, $w$ is searched to mode connected to both $\hat{w}_1$ and $\hat{w}_2$, gradually integrating both models. (Right) After MergeTune Training: Our continued model $w_{ours}$ merging two endpoints will be used for inference of both tasks $\hat{w}_{1}$ and $\hat{w}_{2}$ where trained. The two distinct low-loss paths, $\hat{w}_{1} \rightarrow w_{ours}$ and $\hat{w}_{2} \rightarrow w_{ours}$, show smooth interpolation curves (inset) indicating stable performance.
• Figure 3: Hyperparameter sensitivity analysis: HM averaged over 11 datasets. (a) Surrogate loss weight $\lambda$ and Task2 LMC loss weight $\beta$. (b) Initialisation parameter $\tau$ of the continued model on performance (under $\lambda=8.0$, $\beta=0.5$), where $\tau=0$ means using CLIP weights for initialisation and $\tau=1$ means using the fine-tuned weights (e.g. KgCoOp here).
• Figure 4: Linear mode connectivity analysis on base-to-novel generalisation. We interpolate between our continued model and both the zero-shot CLIP model (orange line, novel-class performance) and the fine-tuned KgCoOp model (blue line, base-class performance). The smooth paths confirm that MergeTune successfully establishes linear mode connectivity with both endpoints, maintaining strong performance throughout interpolation. Results are averaged over 11 datasets.
• Figure 5: MergeTune exhibits no over-merging across extended training (ranging from 10 to 100 epochs). Performance trajectories across all 11 datasets show either stable performance or continuous improvement without degradation. No performance decline validates its robustness against over-merging through our dual-anchored linear mode connectivity objective.