Power Aware Dynamic Reallocation For Inference

TL;DR

Power constraints dominate LLM inference at scale, motivating disaggregation of prefill and decode across GPUs. RAPID integrates static and dynamic asymmetric power caps with phase-specific GPU allocation to preserve $TTFT$ and $TPOT$ SLOs under a fixed budget, implementing a vLLM-based node with disaggregated KV-cache transfer. The approach demonstrates that non-uniform power distribution yields higher goodput per watt, and that dynamic GPU and power reallocation provides the best SLO attainment, with up to a 2x improvement at peak load over static baselines. This work offers a practical, node-scale solution with potential extension to rack-scale deployments and predictive dynamic management for broader LLM inference efficiency. Overall, RAPID combines disaggregation with power-aware scheduling to boost compute-per-watt while maintaining QoS in power-constrained environments.

Abstract

Disaggregation has emerged as a powerful strategy for optimizing large language model (LLM) inference by separating compute-intensive prefill and memory-bound decode phases across specialized GPUs. This separation improves utilization and throughput under fixed hardware capacity. However, as model and cluster scales grow, power, rather than compute, has become the dominant limiter of overall performance and cost efficiency. In this paper, we propose RAPID, a power-aware disaggregated inference framework that jointly manages GPU roles and power budgets to sustain goodput within strict power caps. RAPID utilizes static and dynamic power reallocation in addition to GPU reallocation to improve performance under fixed power bounds. RAPID improves overall performance and application consistency beyond what is achievable in current disaggregation solutions, resulting in up to a 2x improvement in SLO attainment at peak load when compared to a static assignment without an increase in complexity or cost.

Figures (9)

• Figure 1: Goodput results for disaggregation schemes with varying GPUs (4 Prefill, 4 decode [4P4D], 5 prefill, 3 decode [5P3D], and 4 prefill, 4 decode with non-uniform power cap [4P4D]-RAPID) as a function of queries per second (QPS) per GPU. All configurations use a 4800W node power budget. Higher goodput values are better.
• Figure 2: An eight GPU AMD Instinct™ MI300X Platform showing XGMI bandwidth and GPU TDP limits.
• Figure 3: Time series of total GPU power for an uncapped node when running LongBench with a maximum input token size of 8K and QPS/GPU=1.5. Grey line indicates power limit of 4800W for total GPU power.
• Figure 4: (a) Prefill P90 TTFT and (b) Decode P90 TPOT latency as a function of GPU power and batch size. (c) Effectiveness of power cap initiated by amd-smi command.
• Figure 5: (a) SLO attainment with LongBench TTFT = 1 s and TPOT = 40 ms, and (b) SLO attainment with LongBench TTFT = 1 s and TPOT = 25 ms.