Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MT2ST: Adaptive Multi-Task to Single-Task Learning

Dong Liu, Yanxuan Yu

TL;DR

MT2ST addresses the trade-off between generalization in multi-task learning and task-specific precision in single-task learning by introducing an adaptive training framework that transitions from multi-task to single-task optimization. It combines a shared encoder with two transition mechanisms—Diminish (soft, exponential decay of auxiliary losses) and Switch (hard, epoch-based transition to the main task)—to realize early generalization followed by late specialization. The framework is demonstrated across representation learning, transformers, and diffusion models, showing substantial training-time reductions and competitive or improved accuracy, including up to 67% speed-ups in embedding tasks and notable gains in vision and multimodal settings. MT2ST offers a practical, architecture-agnostic pathway for efficient, scalable task-oriented representation learning with potential extensions to vision and multimodal domains.

Abstract

Efficient machine learning (ML) has become increasingly important as models grow larger and data volumes expand. In this work, we address the trade-off between generalization in multi-task learning (MTL) and precision in single-task learning (STL) by introducing the Multi-Task to Single-Task (MT2ST) framework. MT2ST is designed to enhance training efficiency and accuracy in multi-modal tasks, showcasing its value as a practical application of efficient ML.

MT2ST: Adaptive Multi-Task to Single-Task Learning

TL;DR

MT2ST addresses the trade-off between generalization in multi-task learning and task-specific precision in single-task learning by introducing an adaptive training framework that transitions from multi-task to single-task optimization. It combines a shared encoder with two transition mechanisms—Diminish (soft, exponential decay of auxiliary losses) and Switch (hard, epoch-based transition to the main task)—to realize early generalization followed by late specialization. The framework is demonstrated across representation learning, transformers, and diffusion models, showing substantial training-time reductions and competitive or improved accuracy, including up to 67% speed-ups in embedding tasks and notable gains in vision and multimodal settings. MT2ST offers a practical, architecture-agnostic pathway for efficient, scalable task-oriented representation learning with potential extensions to vision and multimodal domains.

Abstract

Efficient machine learning (ML) has become increasingly important as models grow larger and data volumes expand. In this work, we address the trade-off between generalization in multi-task learning (MTL) and precision in single-task learning (STL) by introducing the Multi-Task to Single-Task (MT2ST) framework. MT2ST is designed to enhance training efficiency and accuracy in multi-modal tasks, showcasing its value as a practical application of efficient ML.

Paper Structure

This paper contains 53 sections, 23 equations, 8 figures, 4 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Motivation
  4. Challenges in Single-Task Representation Learning
  5. Data Scale and Cost.
  6. Computational Complexity.
  7. Optimization Challenges.
  8. Improving Training Efficiency via Multi-Task Learning
  9. Shared Representations and Generalization.
  10. Training Efficiency.
  11. Limitations of MTL for General Representation Learning
  12. Gradient Conflicts.
  13. Computational Overhead.
  14. Scalability and Task Imbalance.
  15. Motivating MT2ST: From Multi-Task to Single-Task
  16. ...and 38 more sections

Figures (8)

  • Figure 1: MT2ST Training Framework Overview
  • Figure 2: Loss and Accuracy Change for Single-task Fine-Tuning
  • Figure 3: Loss Change in Multi-task Learning
  • Figure 4: Accuracy Change in Multi-task Learning
  • Figure 5: Loss Change in MT2ST: Diminish Strategy
  • ...and 3 more figures