Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Architecture Search for Sentence Classification with BERT

Philip Kenneweg, Sarah Schröder, Barbara Hammer

TL;DR

This paper questions the common practice of only adding a single output layer as a classification head on top of the network, and performs an AutoML search to find architectures that outperform the current single layer at only a small compute cost.

Abstract

Pre training of language models on large text corpora is common practice in Natural Language Processing. Following, fine tuning of these models is performed to achieve the best results on a variety of tasks. In this paper we question the common practice of only adding a single output layer as a classification head on top of the network. We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost. We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset.

Neural Architecture Search for Sentence Classification with BERT

TL;DR

This paper questions the common practice of only adding a single output layer as a classification head on top of the network, and performs an AutoML search to find architectures that outperform the current single layer at only a small compute cost.

Abstract

Pre training of language models on large text corpora is common practice in Natural Language Processing. Following, fine tuning of these models is performed to achieve the best results on a variety of tasks. In this paper we question the common practice of only adding a single output layer as a classification head on top of the network. We perform an AutoML search to find architectures that outperform the current single layer at only a small compute cost. We validate our classification architecture on a variety of NLP benchmarks from the GLUE dataset.
Paper Structure (7 sections, 1 figure, 4 tables)

This paper contains 7 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Neural Architecture Search Space
  3. Experimental Approach
  4. Datasets
  5. Baseline and Implementation Details
  6. Results
  7. Conclusions

Figures (1)

  • Figure 1: Rough outline of all possible options for the search space of the proposed AutoML pipeline. (Not all options are used all the time)