Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Notes on the Mathematical Structure of GPT LLM Architectures

Spencer Becker-Kahn

Abstract

An exposition of the mathematics underpinning the neural network architecture of a GPT-3-style LLM.

Notes on the Mathematical Structure of GPT LLM Architectures

Abstract

An exposition of the mathematics underpinning the neural network architecture of a GPT-3-style LLM.

Paper Structure

This paper contains 16 sections, 43 equations.

Table of Contents

  1. Tokenization, Encodings and Embeddings
  2. The Vocabulary
  3. Embedding and Unembedding
  4. Feedforward Layers
  5. Basic General Definitions
  6. The Fully Connected Multi-Layer Perceptron
  7. Feedforward Layers
  8. Attention Layers
  9. Softmax with Autoregressive Masking
  10. Attention Heads
  11. Attention Patterns
  12. Attention Layers
  13. The Full Transformer
  14. Residual Blocks
  15. The Decoder Stack
  16. ...and 1 more sections

Theorems & Definitions (6)

  • Remark 1.1
  • Remark 1.2
  • Remark 2.1
  • Remark 2.2
  • Remark 3.1
  • Remark 4.1