Theoretically Optimal Attention/FFN Ratios in Disaggregated LLM Serving

TL;DR

This work addresses efficient inference for disaggregated LLM serving by rigorously sizing the Attention/FFN ratio in an $r$A–$1$F architecture. It develops a probabilistic workload model with continuous batching, derives a horizon-average token load, and yields a closed-form $r^*$ that delineates three operating regimes (Attention-, Communication-, and FFN-bottleneck). A trace-calibrated simulator validates the theory, showing the predicted $r^*$ closely matches the simulation optimum and providing practical guidance for hardware-aware deployment. The findings offer a principled recipe for balancing memory-bound Attention with compute-bound FFN, enabling near-optimal throughput and reduced idling in disaggregated LLM serving.

Abstract

Attention-FFN disaggregation (AFD) is an emerging architecture for LLM decoding that separates state-heavy, KV-cache-dominated Attention computation from stateless, compute-intensive FFN computation, connected by per-step communication. While AFD enables independent scaling of memory and compute resources, its performance is highly sensitive to the Attention/FFN provisioning ratio: mis-sizing induces step-level blocking and costly device idle time. We develop a tractable analytical framework for sizing AFD bundles in an $r$A-$1$F topology, where the key difficulty is that Attention-side work is nonstationary-token context grows and requests are continuously replenished with random lengths-while FFN work is stable given the aggregated batch. Using a probabilistic workload model, we derive closed-form rules for the optimal A/F ratio that maximize average throughput per instance across the system. A trace-calibrated AFD simulator validates the theory: across workloads, the theoretical optimal A/F ratio matches the simulation-optimal within 10%, and consistently reduces idle time.

Figures (8)

• Figure 1: Architecture of AFD. Stateful Attention (A) layers manage the KV cache, while Feed-Forward Network (F) layers are stateless. During each decode step, every continuing request generates one output token whose key-value is appended to the KV cache (red blocks); when a request completes, its slot is immediately refilled with a new prefill request (green block).
• Figure 2: Microbatch Pipelining and Masking. (a) An ideal schedule where Attention, Communication, and FFN are perfectly overlapped across microbatches, fully hiding data transfer latency. (b) After one decode step, Attention execution time increases (due to longer KV cache reads) while Communication and FFN remain unchanged, breaking the balanced overlap and introducing pipeline bubbles.
• Figure 3: Empirical distributions of decode lengths from production LLM traces wang2023openchatwang2024burstgptzheng2023lmsyszhao2024wildchat. Decode lengths exhibit a geometric (discrete-exponential) pattern.
• Figure 4: Per-instance throughput as a function of A/F ratio $r$ with $B=256$, $\mu_D=500$, $\mu_P=100$. Throughput increases with $r$ until reaching the optimal point $r^* \approx 9.3$, after which FFN becomes saturated and throughput per instance decreases.
• Figure 5: Attention idle ratio $\eta_A$ (blue) and FFN idle ratio $\eta_F$ (orange) as functions of A/F ratio $r$. The crossover point where $\eta_A \approx \eta_F$ indicates the balanced configuration.

Theorems & Definitions (5)

• Lemma 4.1: Expected token load
• proof : Proof sketch
• Definition 4.2: Horizon-average token load
• Proposition 4.3: Law of Large Numbers
• Theorem 4.4: Optimal A/F Ratio