Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics

Tian Bowen, Lai Songning, Wu Jiemin, Shuai Zhihao, Ge Shiming, Yue Yutao

TL;DR

A training-free framework designed to enhance multi-task performance through task-specific parameter fusion, STA reduces the need for extensive hyperparameter tuning, thereby improving the generalization and robustness of the model.

Abstract

Pretrained models have revolutionized deep learning by enabling significant performance improvements across a wide range of tasks, leveraging large-scale, pre-learned knowledge representations. However, deploying these models in real-world multi-task learning (MTL) scenarios poses substantial challenges, primarily due to high computational costs and inefficiencies in inference. Traditional approaches such as pruning, quantization, and knowledge distillation have been explored to mitigate these issues, but they often fall short in fully addressing the complexities of multi-task environments. This paper introduces \textbf{\underline{S}}elective \textbf{\underline{T}}ask \textbf{\underline{A}}rithmetic \underline{\textbf{(STA)}}, a training-free framework designed to enhance multi-task performance through task-specific parameter fusion. STA addresses three key challenges: (i) \textbf{Parameter importance diversity: } Recognizing that different tasks relie on distinct parameters, STA employs a loss-sensitive parameter importance metric derived from a first-order Taylor expansion to accurately measure the importance of parameters for each task. (ii) \textbf{Over-reliance on hyperparameter tuning: }By enhancing the sparsity of task vectors through parameter importance metrics, STA reduces the need for extensive hyperparameter tuning, thereby improving the generalization and robustness of the model. (iii) \textbf{Neglect of other abilities in task arithmetic: } Previous works have largely overlooked the potential for more precise task forgetting. STA leverages its parameter importance metric to achieve more controlled and effective task forgetting, minimizing the impact of noisy elements that can degrade model performance. Experimental results demonstrate that STA achieves superior multi-task performance across benchmarks and excellent performance in task forgetting.

Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics

TL;DR

A training-free framework designed to enhance multi-task performance through task-specific parameter fusion, STA reduces the need for extensive hyperparameter tuning, thereby improving the generalization and robustness of the model.

Abstract

Pretrained models have revolutionized deep learning by enabling significant performance improvements across a wide range of tasks, leveraging large-scale, pre-learned knowledge representations. However, deploying these models in real-world multi-task learning (MTL) scenarios poses substantial challenges, primarily due to high computational costs and inefficiencies in inference. Traditional approaches such as pruning, quantization, and knowledge distillation have been explored to mitigate these issues, but they often fall short in fully addressing the complexities of multi-task environments. This paper introduces \textbf{\underline{S}}elective \textbf{\underline{T}}ask \textbf{\underline{A}}rithmetic \underline{\textbf{(STA)}}, a training-free framework designed to enhance multi-task performance through task-specific parameter fusion. STA addresses three key challenges: (i) \textbf{Parameter importance diversity: } Recognizing that different tasks relie on distinct parameters, STA employs a loss-sensitive parameter importance metric derived from a first-order Taylor expansion to accurately measure the importance of parameters for each task. (ii) \textbf{Over-reliance on hyperparameter tuning: }By enhancing the sparsity of task vectors through parameter importance metrics, STA reduces the need for extensive hyperparameter tuning, thereby improving the generalization and robustness of the model. (iii) \textbf{Neglect of other abilities in task arithmetic: } Previous works have largely overlooked the potential for more precise task forgetting. STA leverages its parameter importance metric to achieve more controlled and effective task forgetting, minimizing the impact of noisy elements that can degrade model performance. Experimental results demonstrate that STA achieves superior multi-task performance across benchmarks and excellent performance in task forgetting.

Paper Structure

This paper contains 19 sections, 21 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Task Definition
  4. Task Arithmetic
  5. STA: Our Method
  6. Measure the Importance of Model Parameters
  7. Crossmasking
  8. Task fusion
  9. Task Forgetting
  10. Extension
  11. Task Fusion Experiments
  12. Setup
  13. Dual-task Fusion
  14. Multi-task Fusion
  15. Task Forgetting Experiments
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Our STA method allows for more precise control of task vectors for better task arithmetic.
  • Figure 2: The overview of our proposed Selective Task Arithmetic. In STA, we measure the importance of the task vector through the loss-sensitive parameter importance measurement method, complete the sparsification of the task vector through the importance to obtain the sparse task vector, and then use this task vector to complete tasks such as task fusion or specific task forgetting. A more detailed description of the process is provided in \ref{['sec:method']}.
  • Figure 3: This is a comparison chart of the effect of the two tasks in the case of fusing the two tasks with the task arithmetic method, where the vertical axis represents the accuracy of the test set
  • Figure 4: Visualization on different types of model layers for different tasks