Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BYOM: Building Your Own Multi-Task Model For Free

Weisen Jiang, Baijiong Lin, Han Shi, Yu Zhang, Zhenguo Li, James T. Kwok

TL;DR

BYOM tackles the challenge of building a high-performance multi-task system from many task-specific finetuned checkpoints without access to task data. It introduces two parameter-efficient merging strategies: BYOM-FFT, which injects task knowledge as sparse vectors from fully finetuned models, and BYOM-LoRA, which compresses LoRA factors using a rank-$q$ SVD-based approximation. Across CV and NLP benchmarks, BYOM-FFT and BYOM-LoRA consistently outperform existing merging methods and approach Single-Task performance while using far fewer parameters, with BYOM-FFT also serving as a general add-on to boost other mergers. This data-free, computation-efficient approach broadens practical deployment of multi-task systems and suggests directions for integrating task knowledge into shared backbones.

Abstract

Recently, various merging methods have been proposed to build a multi-task model from task-specific finetuned models without retraining. However, existing methods suffer from a large performance deterioration compared to using multiple task-specific models. In this paper, we propose to inject task-specific knowledge into the merged model and design two parameter-efficient approaches (BYOM-FFT and BYOM-LoRA) to Build Your Own Multi-task model. BYOM-FFT is for merging fully finetuned models, while BYOM-LoRA is for LoRA-finetuned models. Both methods are data-free and computation-efficient. Extensive experiments on computer vision and natural language processing tasks show that the proposed BYOM methods outperform existing merging methods by a large margin. Moreover, BYOM-FFT is general and can be integrated into existing merging methods to further boost performance.

BYOM: Building Your Own Multi-Task Model For Free

TL;DR

BYOM tackles the challenge of building a high-performance multi-task system from many task-specific finetuned checkpoints without access to task data. It introduces two parameter-efficient merging strategies: BYOM-FFT, which injects task knowledge as sparse vectors from fully finetuned models, and BYOM-LoRA, which compresses LoRA factors using a rank- SVD-based approximation. Across CV and NLP benchmarks, BYOM-FFT and BYOM-LoRA consistently outperform existing merging methods and approach Single-Task performance while using far fewer parameters, with BYOM-FFT also serving as a general add-on to boost other mergers. This data-free, computation-efficient approach broadens practical deployment of multi-task systems and suggests directions for integrating task knowledge into shared backbones.

Abstract

Recently, various merging methods have been proposed to build a multi-task model from task-specific finetuned models without retraining. However, existing methods suffer from a large performance deterioration compared to using multiple task-specific models. In this paper, we propose to inject task-specific knowledge into the merged model and design two parameter-efficient approaches (BYOM-FFT and BYOM-LoRA) to Build Your Own Multi-task model. BYOM-FFT is for merging fully finetuned models, while BYOM-LoRA is for LoRA-finetuned models. Both methods are data-free and computation-efficient. Extensive experiments on computer vision and natural language processing tasks show that the proposed BYOM methods outperform existing merging methods by a large margin. Moreover, BYOM-FFT is general and can be integrated into existing merging methods to further boost performance.
Paper Structure (23 sections, 2 equations, 12 figures, 10 tables, 2 algorithms)

This paper contains 23 sections, 2 equations, 12 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Task Interference in Merging Models
  4. Proposed BYOM
  5. Reusing Fully Finetuned Models
  6. Reusing LoRA Finetuned Models
  7. Experiments
  8. Evaluation on CV Tasks
  9. Evaluation on NLP Tasks
  10. Usefulness of Integrating BYOM-FFT into Existing Merging Methods
  11. Effects of $m\%$ on Post-Pruning and BYOM-FFT
  12. Experiments with a CNN-based Model
  13. Effects of $q$ on BYOM-LoRA
  14. Ablation Study on Approximating LoRA Matrices
  15. Conclusion
  16. ...and 8 more sections

Figures (12)

  • Figure 1: Testing accuracy (averaged over eight tasks) of methods merging fully finetuned models.
  • Figure 2: Relative performance with the number of tasks in merging task-specific models fully finetuned from ViT-B/32.
  • Figure 3: Testing accuracy (averaged over two tasks) of merging methods on two dissimilar tasks DTD and Cars.
  • Figure 4: Testing accuracy (averaged over two tasks) of merging methods on two similar tasks split from Cars.
  • Figure 5: Testing accuracy (averaged over eight tasks) of methods reusing LoRA finetuned models.
  • ...and 7 more figures