The New LLM Bottleneck: A Systems Perspective on Latent Attention and Mixture-of-Experts

TL;DR

This paper reframes LLM performance bottlenecks by showing that Multi-head Latent Attention (MLA) and Mixture-of-Experts (MoE) move the bottleneck away from memory-bound attention toward compute-bound or balanced regimes, enabling larger batch sizes during inference.It demonstrates that MLA reduces KV cache size via latent compression and can be ordered to boost data reuse, while MoE scales capacity through sparse routing, requiring large batches to saturate accelerators.An end-to-end analysis of serving systems reveals that high-bandwidth interconnects and data/ expert parallelism are essential to exploit MLA/MoE synergy and meet SLOs, with practical guidance on when to deploy DP versus TP and how to scale across accelerators.Together, these findings argue for a system-centric design methodology that emphasizes compute/memory balance and interconnects over specialized attention hardware.

Abstract

Computational workloads composing traditional Transformer models are starkly bifurcated. Multi-Head Attention (MHA) is memory-bound, with low arithmetic intensity, while feedforward layers are compute-bound. This dichotomy has long motivated research into specialized hardware to mitigate the MHA bottleneck. This paper argues that recent architectural shifts, namely Multi-head Latent Attention (MLA) and Mixture-of-Experts (MoE), challenge the premise of specialized attention hardware. We make two key observations. First, the arithmetic intensity of MLA is over two orders of magnitude greater than that of MHA, shifting it close to a compute-bound regime well-suited for modern accelerators like GPUs. Second, by distributing MoE experts across a pool of accelerators, their arithmetic intensity can be tuned through batching to match that of the dense layers, creating a more balanced computational profile. These findings reveal a diminishing need for specialized attention hardware. The central challenge for next-generation Transformers is no longer accelerating a single memory-bound layer. Instead, the focus must shift to designing balanced systems with sufficient compute, memory capacity, memory bandwidth, and high-bandwidth interconnects to manage the diverse demands of large-scale models.

Figures (12)

• Figure 1: Transformer-decoder-based LLM architecture.
• Figure 2: Computation flow of multi-head latent attention (MLA) with/without layer reordering. ⓐ to ⓗ refer to the layers of MLA (e.g., ⓐ: QKV compression, ⓑ: Q RoPE, ⓒ: Q decompression, ⓓ: K decompression, ⓔ: V decompression, ⓕ: score, ⓖ: K RoPE, ⓗ: context).
• Figure 3: Time per output token and per-device throughput of DeepSeek-R1 and GPT-3 across varying sequence length and batch size. The blurred area indicates configurations where out-of-memory errors occurred and the red dashed line represents the time required to read all data from HBM, which determines the maximum feasible combinations of sequence length and batch size. The experiment assumes a 32 NVIDIA B200 GPU system.
• Figure 4: Memory usage comparison of DeepSeek-R1 and GPT-3, including attention weight, FFN/MoE weight, and the KV$ for 256K tokens with BF16 precision. While GPT-3 needs to use full model parameters during inference, DeepSeek-R1, which employs MoE architecture, activates only 37B parameters per token, requiring approximately 70GB of memory usage.
• Figure 5: (a) Normalized latency of the attention block in the decode stage without reordering compared to a reordered Attention block. (b) and (c) shows execution time ratio of each layer in the attention block in the decode stage with and without reordering, across varying sequence length and batch size. The experiments assume a 32 NVIDIA B200 GPU system.