Deep learning for pedestrians: backpropagation in Transformers

TL;DR

This work derives manual backpropagation for transformer-based next-token-prediction models, extending prior vectorized CNN backprop derivations to embedding, multi-head self-attention, layer normalization, and LoRA layers. It presents a minimal GPT‑like architecture with explicit gradient expressions for all components, including the embedding lookup, attention weights, value/queries/keys pathways, and the LoRA fine-tuning scheme. Key contributions include a detailed backward pass through single- and multi-headed attention, codified complexities, permutation properties, KV caching for autoregressive generation, LN vs BN considerations, and a parameter-efficient LoRA approach. The results offer a tangible, interpretable framework for understanding how forward propagations and losses backpropagate through transformer components, enabling manual analysis and potential optimizations in small to mid-scale GPT-like models.

Abstract

This document is a follow-up to our previous paper dedicated to a vectorized derivation of backpropagation in CNNs. Following the same principles and notations already put in place there, we now focus on transformer-based next-token-prediction architectures. To this end, we apply our lightweight index-free methodology to new types of layers such as embedding, multi-headed self-attention and layer normalization. In addition, we also provide gradient expressions for LoRA layers to illustrate parameter-efficient fine-tuning. Why bother doing manual backpropagation when there are so many tools that do this automatically? Any gap in understanding of how values propagate forward will become evident when attempting to differentiate the loss function. By working through the backward pass manually, we gain a deeper intuition for how each operation influences the final output. A complete PyTorch implementation of a minimalistic GPT-like network is also provided along with analytical expressions for of all of its gradient updates.