Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Attention to Disaggregation: Tracing the Evolution of LLM Inference

Madabattula Rajesh Kumar, Srinivasa Rao Aravilli, Mustafa Saify, Shashank Srivastava

TL;DR

This paper addresses the bottleneck of LLM inference at scale by arguing that traditional monolithic pipelines cannot meet latency, throughput, and cost targets as models reach trillions of parameters. It proposes disaggregated inference—decoupling compute-intensive prefill from memory-bound decode—guided by distributed-systems principles like service decomposition and workload partitioning, to enable independent optimization of TTFT, ITL, and TPOT. The core contributions include a survey of six architectural enhancements (KV Cache, Flash Attention, Speculative Decoding, Continuous Batching, Paged Attention, Radix Attention) and a comparative analysis of three disaggregated framework archetypes (DistServe, AIBrix, NVIDIA Dynamo), covering architecture, resource management, and data-transfer strategies. Collectively, these approaches demonstrate improvements in $TTFT$, $ITL$, and goodput, enabling practical, multi-tenant deployment of very large models with favorable cost-per-query at scale.

Abstract

The evolution of Large Language Models from the Transformer architecture to models with trillions of parameters has shifted the primary bottleneck from model training to real time inference. Deploying these massive models is a complex distributed systems challenge constrained by memory bandwidth, computational throughput, and latency requirements. LLM inference fundamentally requires solving a multi objective optimization problem to minimize latency, maximize throughput, and reduce cost. This paper explores the necessary architectural shift towards disaggregated inference, which applies distributed systems principles such as service decomposition, resource disaggregation, and workload partitioning to overcome the limitations of traditional monolithic GPU clusters. By decoupling the compute intensive prefill phase from the memory intensive decode phase into independently scalable components, this paradigm mitigates resource contention and enables independent optimization of key metrics like Time to First Token and Inter Token Latency.

From Attention to Disaggregation: Tracing the Evolution of LLM Inference

TL;DR

This paper addresses the bottleneck of LLM inference at scale by arguing that traditional monolithic pipelines cannot meet latency, throughput, and cost targets as models reach trillions of parameters. It proposes disaggregated inference—decoupling compute-intensive prefill from memory-bound decode—guided by distributed-systems principles like service decomposition and workload partitioning, to enable independent optimization of TTFT, ITL, and TPOT. The core contributions include a survey of six architectural enhancements (KV Cache, Flash Attention, Speculative Decoding, Continuous Batching, Paged Attention, Radix Attention) and a comparative analysis of three disaggregated framework archetypes (DistServe, AIBrix, NVIDIA Dynamo), covering architecture, resource management, and data-transfer strategies. Collectively, these approaches demonstrate improvements in , , and goodput, enabling practical, multi-tenant deployment of very large models with favorable cost-per-query at scale.

Abstract

The evolution of Large Language Models from the Transformer architecture to models with trillions of parameters has shifted the primary bottleneck from model training to real time inference. Deploying these massive models is a complex distributed systems challenge constrained by memory bandwidth, computational throughput, and latency requirements. LLM inference fundamentally requires solving a multi objective optimization problem to minimize latency, maximize throughput, and reduce cost. This paper explores the necessary architectural shift towards disaggregated inference, which applies distributed systems principles such as service decomposition, resource disaggregation, and workload partitioning to overcome the limitations of traditional monolithic GPU clusters. By decoupling the compute intensive prefill phase from the memory intensive decode phase into independently scalable components, this paradigm mitigates resource contention and enables independent optimization of key metrics like Time to First Token and Inter Token Latency.

Paper Structure

This paper contains 44 sections, 40 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Evolution from Transformers to Large Language Models: A quantum leap in AI
  3. The Inference Bottleneck: Where Scale Meets Reality
  4. Microservices and Heterogeneous GPU Clusters: Where Modularity Meets Efficiency
  5. The Need for Disaggregated Inference: Where Separation Drives Performance
  6. Core Algorithmic and Architectural Enhancements in Large Language Model Inference
  7. KV Cache: Memory Meets Throughput
  8. Flash Attention: High-Throughput Attention via I/O Optimization
  9. Speculative Decoding: Accelerating Inference through Predictive Drafting
  10. Parallel Speculative Decoding : Accelerating Inference through Parallel Adaptive Drafting
  11. Continuous Batching: Maximizing Throughput
  12. Paged Attention: Intelligent Memory Management
  13. Radix Attention : Efficiency Through Sharing
  14. The Fundamental Challenges of LLM Inference Serving
  15. Anatomy of LLM Inference : The Prefill and Decode Duality
  16. ...and 29 more sections

Figures (40)

  • Figure 1: A quantum leap in AI
  • Figure 2: Journey of GPT models
  • Figure 3: Where Scale Meets Reality
  • Figure 4: Where Modularity Meets Efficiency
  • Figure 5: Where Separation Drives Performance
  • ...and 35 more figures