Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models

Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, Dacheng Tao

TL;DR

SMILE addresses the high cost and interference risks of deep model fusion by introducing a zero-shot, sparse mixture of low-rank experts that upscales sources via subspace analysis. Grounded in a subspace decomposition of fine-tuning (via $W= W$ + $ΔW$ and a SVD $W=U Σ V^T$) and a formal interference framework, SMILE uses low-rank adapters extracted through SVD and a routing mechanism $r^{(i)}=\| V^{(i)T}_{k_{gate}} x \|_2$ to selectively combine task-specific updates with a shared pre-trained backbone. It demonstrates strong, scalable performance on open-vocabulary vision tasks and text generation, with competitive accuracy at a fraction of the parameter budget and applicability to large LLMs like Mistral-7B; results show near-parity with fully fine-tuned models at modest overhead and robust behavior across hyperparameters. The work highlights the value of subspace-aware fusion, offering a practical pathway to multi-task and multi-domain models without retraining data, while outlining future extensions such as expanding the expert pool, leveraging LoRA more broadly, and dynamic routing adaptations for efficiency and effectiveness.

Abstract

Deep model training on extensive datasets is increasingly becoming cost-prohibitive, prompting the widespread adoption of deep model fusion techniques to leverage knowledge from pre-existing models. From simple weight averaging to more sophisticated methods like AdaMerging, model fusion effectively improves model performance and accelerates the development of new models. However, potential interference between parameters of individual models and the lack of interpretability in the fusion progress remain significant challenges. Existing methods often try to resolve the parameter interference issue by evaluating attributes of parameters, such as their magnitude or sign, or by parameter pruning. In this study, we begin by examining the fine-tuning of linear layers through the lens of subspace analysis and explicitly define parameter interference as an optimization problem to shed light on this subject. Subsequently, we introduce an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction, which allows for the upscaling of source models into an MoE model without extra data or further training. Our approach relies on the observation that fine-tuning mostly keeps the important parts from the pre-training, but it uses less significant or unused areas to adapt to new tasks. Also, the issue of parameter interference, which is intrinsically intractable in the original parameter space, can be managed by expanding the dimensions. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning, and we apply our method to large language models (CLIP models, Flan-T5 models, and Mistral-7B models), highlighting the adaptability and scalability of SMILE. Code is available at https://github.com/tanganke/fusion_bench

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models

TL;DR

SMILE addresses the high cost and interference risks of deep model fusion by introducing a zero-shot, sparse mixture of low-rank experts that upscales sources via subspace analysis. Grounded in a subspace decomposition of fine-tuning (via + and a SVD ) and a formal interference framework, SMILE uses low-rank adapters extracted through SVD and a routing mechanism to selectively combine task-specific updates with a shared pre-trained backbone. It demonstrates strong, scalable performance on open-vocabulary vision tasks and text generation, with competitive accuracy at a fraction of the parameter budget and applicability to large LLMs like Mistral-7B; results show near-parity with fully fine-tuned models at modest overhead and robust behavior across hyperparameters. The work highlights the value of subspace-aware fusion, offering a practical pathway to multi-task and multi-domain models without retraining data, while outlining future extensions such as expanding the expert pool, leveraging LoRA more broadly, and dynamic routing adaptations for efficiency and effectiveness.

Abstract

Deep model training on extensive datasets is increasingly becoming cost-prohibitive, prompting the widespread adoption of deep model fusion techniques to leverage knowledge from pre-existing models. From simple weight averaging to more sophisticated methods like AdaMerging, model fusion effectively improves model performance and accelerates the development of new models. However, potential interference between parameters of individual models and the lack of interpretability in the fusion progress remain significant challenges. Existing methods often try to resolve the parameter interference issue by evaluating attributes of parameters, such as their magnitude or sign, or by parameter pruning. In this study, we begin by examining the fine-tuning of linear layers through the lens of subspace analysis and explicitly define parameter interference as an optimization problem to shed light on this subject. Subsequently, we introduce an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction, which allows for the upscaling of source models into an MoE model without extra data or further training. Our approach relies on the observation that fine-tuning mostly keeps the important parts from the pre-training, but it uses less significant or unused areas to adapt to new tasks. Also, the issue of parameter interference, which is intrinsically intractable in the original parameter space, can be managed by expanding the dimensions. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning, and we apply our method to large language models (CLIP models, Flan-T5 models, and Mistral-7B models), highlighting the adaptability and scalability of SMILE. Code is available at https://github.com/tanganke/fusion_bench
Paper Structure (16 sections, 2 theorems, 22 equations, 9 figures, 11 tables)

This paper contains 16 sections, 2 theorems, 22 equations, 9 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Rethinking Model Fine-Tuning From a Subspace Perspective
  3. Parameter Interference Between Task-Specific Models
  4. Resolving Parameter Interference using Sparse Mixture of Low-Rank Experts
  5. Experiments
  6. Multi-Task Model Fusion on Open-Vocabulary Image Classification Tasks
  7. Multi-Task Model Fusion on Text Generation Tasks
  8. Spare Mixture of Low-Rank Experts Analysis
  9. Scalability to Large-Scale Models (Mistral-7B models)
  10. Related Work
  11. Conclusion, Limitations, and Future Work
  12. Projecting Fine-tuned Updates onto Subspaces
  13. The Gradient Flow During Fine-tuning
  14. Fine-Tuned Model Performance
  15. Hyperparameter Analysis
  16. ...and 1 more sections

Key Result

Theorem 1

Given two sets of orthonormal vectors $\{u_i\}_{i=1}^{p}\subset \mathbb{R}^m$ and $\{v_i\}_{i=1}^{q}\subset \mathbb{R}^n$, $1 \leq p\leq m$ and $1\leq q\leq n$, the set of matrices $\{u_i v_j^T\}_{i=1,j=1}^{p,q}$ forms an orthonormal basis for a subspace of $\mathbb{R}^{m \times n}$ with dimension $

Figures (9)

  • Figure 1: Multi-task model fusion experiment on eight image classification tasks using CLIP-ViT-B/32 models. Here we set $k_{gate}=16$ and $k$ is varied from 4 to 128 to investigate the trade-off between performance and model size.
  • Figure 2: Here we show the SVD decomposition and subspace partition of the singular value matrix $\Sigma$, and the accuracy comparison of different subspace projection strategies discussed in Section \ref{['sec:revisiting_fine_tuning']}.
  • Figure 3: The architecture of the proposed Sparse MIxture of Low-rank Experts (SMILE) model.
  • Figure 4: Multi-task model fusion experiment on eight image classification tasks using CLIP-ViT-L/14 models ($k_{gate}=16$).
  • Figure 5: Hyperparameter analysis of the Flan-T5-Base models on eight tasks from GLUE benchmark. We show how different values of hyperparameters $k$ and $k_{gate}$ affect the average performance and the normalized number of parameters in the upscaled model. Subfigures (a), and (b) show the results of the full fine-tuned models, while subfigures (c), and (d) show the results of the fine-tuned models with $r_{LoRA}=16$.
  • ...and 4 more figures

Theorems & Definitions (5)

  • Theorem 1
  • Proof 1
  • Remark 1
  • Remark 2
  • Theorem 2