Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformer models: an introduction and catalog

Xavier Amatriain, Ananth Sankar, Jie Bing, Praveen Kumar Bodigutla, Timothy J. Hazen, Michaeel Kazi

TL;DR

The paper addresses the rapid proliferation of Transformer foundation models and provides a practical catalog. It develops a taxonomy based on pretraining architecture, tasks, and extensions, plus a family tree and timeline. It also offers background on core Transformer concepts (encoder/decoder, attention), foundation vs fine-tuned models, and notes on diffusion models. This catalog supports researchers and practitioners in navigating model choices for downstream tasks and tracking historical and architectural trends.

Abstract

In the past few years we have seen the meteoric appearance of dozens of foundation models of the Transformer family, all of which have memorable and sometimes funny, but not self-explanatory, names. The goal of this paper is to offer a somewhat comprehensive but simple catalog and classification of the most popular Transformer models. The paper also includes an introduction to the most important aspects and innovations in Transformer models. Our catalog will include models that are trained using self-supervised learning (e.g., BERT or GPT3) as well as those that are further trained using a human-in-the-loop (e.g. the InstructGPT model used by ChatGPT).

Transformer models: an introduction and catalog

TL;DR

The paper addresses the rapid proliferation of Transformer foundation models and provides a practical catalog. It develops a taxonomy based on pretraining architecture, tasks, and extensions, plus a family tree and timeline. It also offers background on core Transformer concepts (encoder/decoder, attention), foundation vs fine-tuned models, and notes on diffusion models. This catalog supports researchers and practitioners in navigating model choices for downstream tasks and tracking historical and architectural trends.

Abstract

In the past few years we have seen the meteoric appearance of dozens of foundation models of the Transformer family, all of which have memorable and sometimes funny, but not self-explanatory, names. The goal of this paper is to offer a somewhat comprehensive but simple catalog and classification of the most popular Transformer models. The paper also includes an introduction to the most important aspects and innovations in Transformer models. Our catalog will include models that are trained using self-supervised learning (e.g., BERT or GPT3) as well as those that are further trained using a human-in-the-loop (e.g. the InstructGPT model used by ChatGPT).
Paper Structure (95 sections, 8 figures)

This paper contains 95 sections, 8 figures.

Table of Contents

  1. Introduction: What are Transformers
  2. Encoder/Decoder architecture
  3. Attention
  4. Foundation vs Fine-tuned models
  5. The Impact of Transformers
  6. A Note on Diffusion models
  7. The Transformers Catalog
  8. Features of a Transformer
  9. Pretraining Architecture
  10. Encoder Pretraining
  11. Decoder Pretraining
  12. Transformer (Encoder-Decoder) Pretraining
  13. Pretraining or Finetuning Task
  14. Application
  15. Catalog table
  16. ...and 80 more sections

Figures (8)

  • Figure 1: Transformer Architecture from ( ? )
  • Figure 2: The Attention Mechanism from ( ? ) . (left) Scaled Dot-Product Attention, (right) Multi-Head Attention
  • Figure 3: Reinforcement Learning with Human Feedback. From HuggingFace’s RLHF blog post at https://huggingface.co/blog/rlhf
  • Figure 4: Probabilistic diffusion model architecture from “Diffusion Models: A Comprehensive Survey of Methods and Applications," Figure 2 ( ? )
  • Figure 5: Transformers Family Tree
  • ...and 3 more figures