PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning

TL;DR

PEMT tackles the inefficiency and limited cross-task exploitation of existing PEFT methods by introducing a two-stage, multi-task transfer framework. It trains task-specific adapters and task description prompts on multiple source tasks, then freezes these sources and uses a correlation-guided MoE to fuse source knowledge for a target task, complemented by a target adapter and a sparsity-focused loss. The approach achieves consistent improvements over full fine-tuning and state-of-the-art PEFT methods across 17 NLP datasets, with notable gains in few-shot scenarios and interpretable source-adapter usage patterns. The work advances practical parameter-efficient transfer learning by explicitly modeling task correlations and leveraging task-specific representations to maximize knowledge reuse across tasks.

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as an effective method for adapting pre-trained language models to various tasks efficiently. Recently, there has been a growing interest in transferring knowledge from one or multiple tasks to the downstream target task to achieve performance improvements. However, current approaches typically either train adapters on individual tasks or distill shared knowledge from source tasks, failing to fully exploit task-specific knowledge and the correlation between source and target tasks. To overcome these limitations, we propose PEMT, a novel parameter-efficient fine-tuning framework based on multi-task transfer learning. PEMT extends the mixture-of-experts (MoE) framework to capture the transferable knowledge as a weighted combination of adapters trained on source tasks. These weights are determined by a gated unit, measuring the correlation between the target and each source task using task description prompt vectors. To fully exploit the task-specific knowledge, we also propose the Task Sparsity Loss to improve the sparsity of the gated unit. We conduct experiments on a broad range of tasks over 17 datasets. The experimental results demonstrate our PEMT yields stable improvements over full fine-tuning, and state-of-the-art PEFT and knowledge transferring methods on various tasks. The results highlight the effectiveness of our method which is capable of sufficiently exploiting the knowledge and correlation features across multiple tasks.

Figures (4)

• Figure 1: Performance of different parameter-efficient fine-tuning approaches. All results are based on T5-base for a fair comparison. The proposed PEMT achieves significant improvements over all compared methods while fine-tuning only a small number of parameters.
• Figure 2: The training process of Stage 1. The task-specific adapters and task representation prompts are trained on multiple source tasks.
• Figure 3: The training process of Stage 2. A MoE module is employed to distill knowledge from source tasks. The source task adapters are used as the experts and combined with a MoE gate which measures the correlation between the target task and each source task. The specific adapter for the target task is injected after the MoE module. The task sparsity loss $\mathcal{L}_{ts}$ is incorporated to improve the sparsity of the MoE gate. The task prompt for the target task is a task-correlation-based combination of the trainable prompt vectors and the frozen prompts of the source tasks.
• Figure 4: The source expert weight distribution in GLUE, SuperGLUE, MRQA and Others benchmarks.