Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation

Xiao Zhang, Kangsheng Wang, Tianyu Hu, Huimin Ma

TL;DR

TA-LoRA addresses cross-task knowledge transfer in multi-task learning by enriching prompt tuning with a low-rank representation to model task heterogeneity. A slow (shared) versus fast (task-specific) weight mechanism and a zero-initialized attention stabilize training and prevent interference with the original prompts. The method delivers state-of-the-art performance on 16 NLP tasks in both full-data and few-shot settings while maintaining excellent parameter efficiency, demonstrating robust generalization to unseen data and unseen tasks. By explicitly decoupling shared knowledge from task-specific nuances, TA-LoRA offers a practical, scalable approach for multi-source task adaptation in PLMs.

Abstract

Pre-trained language models (PLMs) demonstrate remarkable intelligence but struggle with emerging tasks unseen during training in real-world applications. Training separate models for each new task is usually impractical. Multi-task learning (MTL) addresses this challenge by transferring shared knowledge from source tasks to target tasks. As an dominant parameter-efficient fine-tuning method, prompt tuning (PT) enhances MTL by introducing an adaptable vector that captures task-specific knowledge, which acts as a prefix to the original prompt that preserves shared knowledge, while keeping PLM parameters frozen. However, PT struggles to effectively capture the heterogeneity of task-specific knowledge due to its limited representational capacity. To address this challenge, we propose Task-Adaptive Low-Rank Representation (TA-LoRA), an MTL method built on PT, employing the low-rank representation to model task heterogeneity and a fast-slow weights mechanism where the slow weight encodes shared knowledge, while the fast weight captures task-specific nuances, avoiding the mixing of shared and task-specific knowledge, caused by training low-rank representations from scratch. Moreover, a zero-initialized attention mechanism is introduced to minimize the disruption of immature low-rank components on original prompts during warm-up epochs. Experiments on 16 tasks demonstrate that TA-LoRA achieves state-of-the-art performance in full-data and few-shot settings while maintaining superior parameter efficiency.

Efficient Knowledge Transfer in Multi-Task Learning through Task-Adaptive Low-Rank Representation

TL;DR

TA-LoRA addresses cross-task knowledge transfer in multi-task learning by enriching prompt tuning with a low-rank representation to model task heterogeneity. A slow (shared) versus fast (task-specific) weight mechanism and a zero-initialized attention stabilize training and prevent interference with the original prompts. The method delivers state-of-the-art performance on 16 NLP tasks in both full-data and few-shot settings while maintaining excellent parameter efficiency, demonstrating robust generalization to unseen data and unseen tasks. By explicitly decoupling shared knowledge from task-specific nuances, TA-LoRA offers a practical, scalable approach for multi-source task adaptation in PLMs.

Abstract

Pre-trained language models (PLMs) demonstrate remarkable intelligence but struggle with emerging tasks unseen during training in real-world applications. Training separate models for each new task is usually impractical. Multi-task learning (MTL) addresses this challenge by transferring shared knowledge from source tasks to target tasks. As an dominant parameter-efficient fine-tuning method, prompt tuning (PT) enhances MTL by introducing an adaptable vector that captures task-specific knowledge, which acts as a prefix to the original prompt that preserves shared knowledge, while keeping PLM parameters frozen. However, PT struggles to effectively capture the heterogeneity of task-specific knowledge due to its limited representational capacity. To address this challenge, we propose Task-Adaptive Low-Rank Representation (TA-LoRA), an MTL method built on PT, employing the low-rank representation to model task heterogeneity and a fast-slow weights mechanism where the slow weight encodes shared knowledge, while the fast weight captures task-specific nuances, avoiding the mixing of shared and task-specific knowledge, caused by training low-rank representations from scratch. Moreover, a zero-initialized attention mechanism is introduced to minimize the disruption of immature low-rank components on original prompts during warm-up epochs. Experiments on 16 tasks demonstrate that TA-LoRA achieves state-of-the-art performance in full-data and few-shot settings while maintaining superior parameter efficiency.
Paper Structure (15 sections, 10 equations, 3 figures, 4 tables)

This paper contains 15 sections, 10 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Problem Formulation
  5. Low-rank Heterogeneity Representation
  6. Slow-Fast Weights Mechanism
  7. Zero-Initialized Attention
  8. Source Optimization and Target Generalization
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Ablation Studies
  13. Conclusion
  14. Interlayer Similarity
  15. Dataset Details

Figures (3)

  • Figure 1: The framework of TA-LoRA serves as a plugin in the final $L$ layers of the PLM. "Repr." and "attn." stand for representation and attention, respectively.
  • Figure 2: Decomposition of the slow weight $\boldsymbol{B}$ and the fast weight $\boldsymbol{A}_i$ for $\boldsymbol{T}_i$, with further decomposition of fast weight $\boldsymbol{A}_i$ into $\boldsymbol{u}_i$ and $\boldsymbol{v}_i$. "lr" stands for the learning rate.
  • Figure 3: The similarity between base models in different layers obtained by PT on AFQMC.