Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recurrent Neural Network for Text Classification with Multi-Task Learning

Pengfei Liu, Xipeng Qiu, Xuanjing Huang

TL;DR

The paper tackles data scarcity in text classification by introducing three RNN-based multi-task architectures that share information across related tasks. It formalizes uniform-, coupled-, and shared-layer sharing with gating mechanisms and trains them jointly, with optional pretraining of the shared layer. Across four datasets, the multi-task models outperform single-task baselines and show competitive gains compared to state-of-the-art neural models, especially when fine-tuned or pre-trained. The work demonstrates that carefully designed cross-task sharing in RNNs can enhance text representations and transfer learning in NLP.

Abstract

Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, in most previous works, the models are learned based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we use the multi-task learning framework to jointly learn across multiple related tasks. Based on recurrent neural network, we propose three different mechanisms of sharing information to model text with task-specific and shared layers. The entire network is trained jointly on all these tasks. Experiments on four benchmark text classification tasks show that our proposed models can improve the performance of a task with the help of other related tasks.

Recurrent Neural Network for Text Classification with Multi-Task Learning

TL;DR

The paper tackles data scarcity in text classification by introducing three RNN-based multi-task architectures that share information across related tasks. It formalizes uniform-, coupled-, and shared-layer sharing with gating mechanisms and trains them jointly, with optional pretraining of the shared layer. Across four datasets, the multi-task models outperform single-task baselines and show competitive gains compared to state-of-the-art neural models, especially when fine-tuned or pre-trained. The work demonstrates that carefully designed cross-task sharing in RNNs can enhance text representations and transfer learning in NLP.

Abstract

Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, in most previous works, the models are learned based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we use the multi-task learning framework to jointly learn across multiple related tasks. Based on recurrent neural network, we propose three different mechanisms of sharing information to model text with task-specific and shared layers. The entire network is trained jointly on all these tasks. Experiments on four benchmark text classification tasks show that our proposed models can improve the performance of a task with the help of other related tasks.

Paper Structure

This paper contains 25 sections, 9 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Recurrent Neural Network for Specific-Task Text Classification
  3. Recurrent Neural Network
  4. Task-Specific Output Layer
  5. Three Sharing Models for RNN based Multi-Task Learning
  6. Model-I: Uniform-Layer Architecture
  7. Model-II: Coupled-Layer Architecture
  8. Model-III: Shared-Layer Architecture
  9. Training
  10. Fine Tuning
  11. Pre-training of the shared layer with neural language model
  12. Experiment
  13. Datasets
  14. Hyperparameters and Training
  15. Effect of Multi-task Training
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Recurrent Neural Network for Classification
  • Figure 2: Three architectures for modelling text with multi-task learning.
  • Figure 3: (a)(b) The change of the predicted sentiment score at different time steps. Y-axis represents the sentiment score, while X-axis represents the input words in chronological order. The red horizontal line gives a border between the positive and negative sentiments. (c)(d) Visualization of the global gate's ($g^{(s)}$) activation.