Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Task Learning in Natural Language Processing: An Overview

Shijie Chen, Yu Zhang, Qiang Yang

TL;DR

This paper surveys multi-task learning (MTL) in NLP as a solution to data scarcity and overfitting, detailing architecture classes (parallel, hierarchical, modular, and generative adversarial) and optimization techniques (loss construction, gradient regularization, data sampling, and task scheduling). It distinguishes auxiliary from joint MTL and discusses multilingual and multimodal extensions, supported by benchmarks such as GLUE, SuperGLUE, XGLUE, and LSParD. The review also covers how task relatedness influences MTL gains, and emphasizes practical aspects like adapters and prefix-tuning for efficient sharing. Overall, thework highlights how carefully designed MTl architectures and training strategies can yield robust, data-efficient NLP systems and points to future directions in task selection, unsupervised data utilization, and expanding MTl to broader NLP tasks and modalities.

Abstract

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Multi-Task Learning in Natural Language Processing: An Overview

TL;DR

This paper surveys multi-task learning (MTL) in NLP as a solution to data scarcity and overfitting, detailing architecture classes (parallel, hierarchical, modular, and generative adversarial) and optimization techniques (loss construction, gradient regularization, data sampling, and task scheduling). It distinguishes auxiliary from joint MTL and discusses multilingual and multimodal extensions, supported by benchmarks such as GLUE, SuperGLUE, XGLUE, and LSParD. The review also covers how task relatedness influences MTL gains, and emphasizes practical aspects like adapters and prefix-tuning for efficient sharing. Overall, thework highlights how carefully designed MTl architectures and training strategies can yield robust, data-efficient NLP systems and points to future directions in task selection, unsupervised data utilization, and expanding MTl to broader NLP tasks and modalities.

Abstract

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Paper Structure

This paper contains 28 sections, 20 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. MTL Architectures for NLP Tasks
  3. Parallel Architectures
  4. Parallel Feature Sharing.
  5. Parallel Feature Fusion
  6. Parallel Multi-level Supervision.
  7. Hierarchical Architectures
  8. Hierarchical Feature Fusion
  9. Hierarchical Pipeline
  10. Hierarchical Interactive MTL
  11. Modular Architectures
  12. Generative Adversarial Architectures
  13. Optimization for MTL Models
  14. Loss Construction
  15. Gradient Regularization
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Illustration for parallel architectures. For task $t$, $h_t^{(i)}$ represents the latent representation at the $i$-th layer and $y_t$ represents the corresponding label ($h_s$ are shared latent representations). The green blocks represent shared parameters and the orange blocks are task-specific parameters. Red circles represent feature fusion mechanism $f$.
  • Figure 2: Illustration for hierarchical architectures. $h$ represents different hidden states and $\hat{y}_t$ represents the predicted output distribution for task $t$. Red boxes stand for hierarchical feature fusion mechanisms. The purple block and blue circle in (b) stand for hierarchical feature and signal pipeline unit respectively.
  • Figure 3: Illustration for multi-task adapters.