LLM Architecture, Scaling Laws, and Economics: A Quick Summary

TL;DR

This work surveys how the standard QKV Transformer architecture, its compute and memory demands, and associated economic implications scale for state-of-the-art LLMs. It consolidates key formulas for forward and training costs, including how autoregressive generation uses caching and how the optimal training set size $D^* ∼ 240 L d_{emb}^2 + 40 V d_{emb}$ tokens emerges from the scaling laws. The analysis is illustrated with numerical examples and a DeepSeek case, showing that MoE-based designs do not violate scaling laws but optimize price/performance through training/inference trade-offs. The results inform architecture design and cost planning for future LLMs, highlighting the practical impact of context length, embedding dimension, and parameter counts on real-world deployments.

Abstract

The current standard architecture of Large Language Models (LLMs) with QKV self-attention is briefly summarized, including the architecture of a typical Transformer. Scaling laws for compute (flops) and memory (parameters plus data) are given, along with their present (2025) rough cost estimates for the parameters of present LLMs of various scales, including discussion of whether DeepSeek should be viewed as a special case. Nothing here is new, but this material seems not otherwise readily available in summary form.

Figures (3)

• Figure 1: Typical LLM architecture (minus bells and whistles). Input is a sequence of $n$ tokens in a vocabulary $V$, encoded one-hot. Matrix multiplication (here denoted $\otimes$) by an embedding matrix $\mathbf{E}$ produces a corresponding sequence of embeddings. These go through multiple transformer layers. In inference, the last resultant embedding (a single row) is de-embedded into $V$ logits, thence probabilities that may be sampled to produce a predicted next-token. The entire chain is trained simultaneously.
• Figure 2: Three components of a typical transformer architecture are a skip conection, an attention function (Figure \ref{['fig:attention']}), and a feed-forward network. The latter has two fully connected layers separated by a nonlinear activation and is applied separately to each of the $n$ input token embeddings.
• Figure 3: The attention function. $H$ attention heads each learn three independent projections from ${d_\text{emb}}$ to $d_h$ dimensions. These are combined in each head as $\mathbf{Q}\mathbf{K}^T\mathbf{V}$ with a softmax and causal masking (see text). The heads' outputs are concatenated and mapped to output by a fully connected linear layer $\mathbf{O}$.